Redux Blog

Last week we deployed a couple of updates to the Dashboard for Virtual Server customers. First off, you can now completely reset your VM back to it's original state. Note: This is not reversable.

You can access this feature by clicking on "Manage" next to your VM, and then clicking "Reset VM".

We also updated our Server Images, and you can now order Centos 6!

I wish this article had been published prior to engaging our first series of SRX units. Over the last couple of months we’ve actively deployed close to (20) SRX 240 units, primarily in active/passive chassis clusters.

Quite a few of our customers rely on VPN connectivity to manage their private clouds. Quite a few of our clients are also Mac shops, similar to our organization.

A popular, yet expensive VPN client is made by our friends at Equinux called VPN Tracker. This is without question the most powerful and robust VPN client I’ve ever seen on any platform be it Mac, Windows or Linux.

That being said, during a transition from Watchguard to Juniper as our Security gateway provider (and Force10 to Juniper MX for Routing) a poor assumption made on our part was that the SRX series was supported out of the box by VPN Tracker. This article is being written because it was, and currently is not supported.

For anyone following the Juniper Security track, it is clear from the writers of JunOS Security by O’Reilly that the SRX is being geared to replace the Netscreen and SSG families. ScreenOS will be no more, JunOS is the future for security at Juniper.

Through a combination of many resources including but not limited to Juniper TAC, a third party site to site (not dynamic VPN) windows based IPSec client (which had documented support by JunOS) and a great deal of troubleshooting I was able to engineer a working configuration for IPSec connectivity from VPN Tracker to an SRX endpoint.

Lets start by saying the standard proposal set should not be used. This was my first glimmer into not using default sets within JunOS. This essentially predefines a set of criteria to be used for the proposal to have a conversation and engage with the endpoint. Specifically:

standard - Standard set of two set of IKE proposals: Proposal 1— Preshared key, 3DES encryption, and Diffie-Hellman Group 2 and SHA-1 authentication. Proposal 2—Preshared key, Advanced Encryption Standard (AES) 128-bit encryption, and Diffie-Hellman Group 2 and SHA-1 authentication.

Despite all efforts we could not engage in a Proposal 1 agreement within VPN Tracker, initial theory is on the Triple DES encryption. What follows is a documented and working proposal set to use for your Dynamic VPN IPSec configurations within JunOS:

set security zones security-zone Public interfaces ge-X/X/X.X host-inbound-traffic system-services ike
set security ike proposal PSK-AES128-SHA1-DH2 authentication-method pre-shared-keys
set security ike proposal PSK-AES128-SHA1-DH2 dh-group group2
set security ike proposal PSK-AES128-SHA1-DH2 authentication-algorithm sha1
set security ike proposal PSK-AES128-SHA1-DH2 encryption-algorithm aes-128-cbc
set security ike proposal PSK-AES128-SHA1-DH2 lifetime-seconds 28800
set security ike proposal PSK-AES256-SHA1-DH2 authentication-method pre-shared-keys
set security ike proposal PSK-AES256-SHA1-DH2 dh-group group2
set security ike proposal PSK-AES256-SHA1-DH2 authentication-algorithm sha1
set security ike proposal PSK-AES256-SHA1-DH2 encryption-algorithm aes-256-cbc
set security ike proposal PSK-AES256-SHA1-DH2 lifetime-seconds 28800
set security ike policy ike-dyn-vpn-policy mode aggressive
set security ike policy ike-dyn-vpn-policy proposals PSK-AES128-SHA1-DH2
set security ike policy ike-dyn-vpn-policy proposals PSK-AES256-SHA1-DH2
set security ike policy ike-dyn-vpn-policy pre-shared-key ascii-text "XXXX"
set security ike gateway dyn-vpn-local-gw ike-policy ike-dyn-vpn-policy
set security ike gateway dyn-vpn-local-gw dynamic hostname reduxftw
set security ike gateway dyn-vpn-local-gw dynamic connections-limit 10
set security ike gateway dyn-vpn-local-gw dynamic ike-user-type shared-ike-id
set security ike gateway dyn-vpn-local-gw external-interface ge-X/X/X.X
set security ike gateway dyn-vpn-local-gw xauth access-profile dyn-vpn-access-profile
set security ipsec proposal ESP-AES128-SHA protocol esp
set security ipsec proposal ESP-AES128-SHA authentication-algorithm hmac-sha1-96
set security ipsec proposal ESP-AES128-SHA encryption-algorithm aes-128-cbc
set security ipsec proposal ESP-AES128-SHA lifetime-seconds 28800
set security ipsec proposal ESP-AES256-SHA protocol esp
set security ipsec proposal ESP-AES256-SHA authentication-algorithm hmac-sha1-96
set security ipsec proposal ESP-AES256-SHA encryption-algorithm aes-256-cbc
set security ipsec proposal ESP-AES256-SHA lifetime-seconds 28800
set security ipsec policy ipsec-dyn-vpn-policy perfect-forward-secrecy keys group2
set security ipsec policy ipsec-dyn-vpn-policy proposals ESP-AES128-SHA
set security ipsec policy ipsec-dyn-vpn-policy proposals ESP-AES256-SHA
set security ipsec vpn dyn-vpn ike gateway dyn-vpn-local-gw
set security ipsec vpn dyn-vpn ike ipsec-policy ipsec-dyn-vpn-policy

Pretty basic configuration, fill in the X’s where needed and you’ve got viable usage of AES128 or 256.

On the VPN Tracker side simply create a custom profile and fill in your phase 1 and phase 2 settings as appropriate.

Also, don’t forget to insert your Public_Bound source bound natting in front of your last resort source-address with source-nat off.

Lastly, remember that security policies in JunOS flow downward. Ensure your security policy to handle this traffic is in front of any-permit.

Hopefully this helps the next engineer working with similar platforms.

Any questions, I'm thomas@networkredux.com. Cheers.

A client recently asked us to recommend a proxy (layer 4 and layer 7) that would scale better than the appliance they were using without breaking the bank. I didn't hesitate to point them to haproxy -- it's fast, stable, scalable, widely deployed and well-supported.

Not long after deploying a pair of Dell R610s running haproxy and heartbeat, the client (wisely) initiated some benchmark testing to evaluate the capabilities of their new proxy layer. The results fell far short of their expectations and ours. A pool of several servers only managed 600 -- 700 sessions per second. Worse yet, running the benchmark directly against one server (bypassing the proxy layer) yielded better results, around 700 -- 800 sessions per second. It's a sad day when a single server outperforms a cluster. What could be holding things back?

I first tried to reproduce the inadequate performance in my dev environment. And although the client was using some just-deployed XenServer 6.0 chassis, my dev environment is still on 5.6. But the problem was easily reproduced despite this difference (unfortunately showing that 6.0 has this same capacity limit as 5.6). My test case used a static document of about 10KB, served under Apache 2. The benchmarks were run with ab (from the Apache package), httperf, and siege using various concurrencies and durations. I wasn't overly impressed with httperf, ab got the job done pretty well, and I found siege to work beautifully. Regardless of my personal evaluations of these tools, though, I found that all three agreed fairly closely on the performance capabilities of my test VM: 600 -- 800 sessions per second, depending on the test parameters. In general, low concurrencies yieled moderately acceptable session service rates, but anything above about 50 -- 100 concurrent sessions caused the overall session rate to plunge. Given that the client was starting their tests at 250 concurrent sessions and moving up from there (hoping for at least 1000/s service), this would never suffice.

While running benchmark after benchmark, I poked at various system statics on the VMs involved and saw nothing useful: haproxy running at about 10% CPU (not terrible given that the network driver was for a generic software "NIC"), Apache was peaking at around 30% CPU, iostat and vmstat showing nothing that would limit the service performance. So I poked on the chassis hosting haproxy and found a process eating 70% of the dom0 CPU capacity -- netback. The software network stack in XenServer itself was unable to handle the load the benchmark was generating.

To show definitively that the proxies were not contributing to the problem in any substantial way, I reworked my test setup on the deployed R610 proxies (which were not yet live due to the performance issues). Using the same static test file, Apache 2, and haproxy configuration, sending traffic from one R610 to the other (the latter hosting both Apache and haproxy), I pulled a cool 10000 sessions per second. More than a ten-fold increase! Watching the processes in top(1), I noted that the reported CPU loads were very volatile during a benchmark run. Pinning the processes to specific CPUs helped here. Only one haproxy process was running (despite the R610 having 8 CPU cores), so I pinned it to a single core, Apache to another single core, and top to a third core. This allowed haproxy to break 11000 sessions per second. That's pretty respectable for a 10KB document on a 1Gbe network.

With a spread from 800/s to 10000/s, it's clear that the overhead of the virtualized network is a serious bottleneck. Optimal performance demands minimal overhead!

bks

Keeping our clients and readers up to date has been difficult these last few months given some significant growth within our organization.

Where to begin, well in the last two months the following has occurred:

• Network Redux has grown its revenue base by 20% in the last 8 weeks.
• We have doubled our capacity in Seattle / Sabey.
• Formally launched datacenter presence in New York City with Internap (NYM008).
• With our launch in NYC we are offering coast to coast SAN replication for Enterprise Clients.
• Marked our 7th year of business as of September 18th.
• Made a significant change in vendors for routing and security by consolidating these requirements into Juniper's MX and SRX series.
• Now hosting a local and well known www.webtrends.com
• Deployed significant changes and feature enhancements to our user Dashboard.

All of this fruit coming to bear while maintaining 100% ownership of our organization.

I am extremely proud of our talented team and welcome in our 8th year of business.

Cheers,

Thomas Brenneke, President Network Redux, LLC

Good news for our readers, followers and customers. We are actively hiring additional Devop oriented folks at our Portland, Oregon based HQ.

Those apply for these positions should find comfort in the following platforms:

• Ruby / Perl
• MySQL / Postgresql
• Mongo / Redis / Riak
• RHEL / CentOS / Ubuntu
• Citrix XenServer
• JunOS

This is a W-2 full time position working from our Portland offices in John's Landing.

With hard work comes many rewards, including but not limited to:

• Comprehensive medical, vision and dental insurance
• Flex-time
• Parking / Commute expenses covered
• Fault tolerant whiteboards (n+1)
• Organizational performance based bonuses

Please send messages directly to myself [thomas] @ {networkredux}.com.