Redux Blog

Virtual Network Performance in Citrix XenServer

(January 11, 2012, Posted in Scalability by Brian Shore)

A client recently asked us to recommend a proxy (layer 4 and layer 7) that would scale better than the appliance they were using without breaking the bank. I didn't hesitate to point them to haproxy -- it's fast, stable, scalable, widely deployed and well-supported.

Not long after deploying a pair of Dell R610s running haproxy and heartbeat, the client (wisely) initiated some benchmark testing to evaluate the capabilities of their new proxy layer. The results fell far short of their expectations and ours. A pool of several servers only managed 600 -- 700 sessions per second. Worse yet, running the benchmark directly against one server (bypassing the proxy layer) yielded better results, around 700 -- 800 sessions per second. It's a sad day when a single server outperforms a cluster. What could be holding things back?

I first tried to reproduce the inadequate performance in my dev environment. And although the client was using some just-deployed XenServer 6.0 chassis, my dev environment is still on 5.6. But the problem was easily reproduced despite this difference (unfortunately showing that 6.0 has this same capacity limit as 5.6). My test case used a static document of about 10KB, served under Apache 2. The benchmarks were run with ab (from the Apache package), httperf, and siege using various concurrencies and durations. I wasn't overly impressed with httperf, ab got the job done pretty well, and I found siege to work beautifully. Regardless of my personal evaluations of these tools, though, I found that all three agreed fairly closely on the performance capabilities of my test VM: 600 -- 800 sessions per second, depending on the test parameters. In general, low concurrencies yieled moderately acceptable session service rates, but anything above about 50 -- 100 concurrent sessions caused the overall session rate to plunge. Given that the client was starting their tests at 250 concurrent sessions and moving up from there (hoping for at least 1000/s service), this would never suffice.

While running benchmark after benchmark, I poked at various system statics on the VMs involved and saw nothing useful: haproxy running at about 10% CPU (not terrible given that the network driver was for a generic software "NIC"), Apache was peaking at around 30% CPU, iostat and vmstat showing nothing that would limit the service performance. So I poked on the chassis hosting haproxy and found a process eating 70% of the dom0 CPU capacity -- netback. The software network stack in XenServer itself was unable to handle the load the benchmark was generating.

To show definitively that the proxies were not contributing to the problem in any substantial way, I reworked my test setup on the deployed R610 proxies (which were not yet live due to the performance issues). Using the same static test file, Apache 2, and haproxy configuration, sending traffic from one R610 to the other (the latter hosting both Apache and haproxy), I pulled a cool 10000 sessions per second. More than a ten-fold increase! Watching the processes in top(1), I noted that the reported CPU loads were very volatile during a benchmark run. Pinning the processes to specific CPUs helped here. Only one haproxy process was running (despite the R610 having 8 CPU cores), so I pinned it to a single core, Apache to another single core, and top to a third core. This allowed haproxy to break 11000 sessions per second. That's pretty respectable for a 10KB document on a 1Gbe network.

With a spread from 800/s to 10000/s, it's clear that the overhead of the virtualized network is a serious bottleneck. Optimal performance demands minimal overhead!

-- bks

Time for a much needed update

(September 29, 2011, Posted in General News by Thomas Brenneke)

Keeping our clients and readers up to date has been difficult these last few months given some significant growth within our organization.

Where to begin, well in the last two months the following has occurred:

• Network Redux has grown its revenue base by 20% in the last 8 weeks.
• We have doubled our capacity in Seattle / Sabey.
• Formally launched datacenter presence in New York City with Internap (NYM008).
• With our launch in NYC we are offering coast to coast SAN replication for Enterprise Clients.
• Marked our 7th year of business as of September 18th.
• Made a significant change in vendors for routing and security by consolidating these requirements into Juniper's MX and SRX series.
• Now hosting a local and well known www.webtrends.com
• Deployed significant changes and feature enhancements to our user Dashboard.

All of this fruit coming to bear while maintaining 100% ownership of our organization.

I am extremely proud of our talented team and welcome in our 8th year of business.

Cheers,

Thomas Brenneke, President Network Redux, LLC

Hiring Devops

(August 22, 2011, Posted in General News by Thomas Brenneke)

Good news for our readers, followers and customers. We are actively hiring additional Devop oriented folks at our Portland, Oregon based HQ.

Those apply for these positions should find comfort in the following platforms:

• Ruby / Perl
• MySQL / Postgresql
• Mongo / Redis / Riak
• RHEL / CentOS / Ubuntu
• Citrix XenServer
• JunOS

This is a W-2 full time position working from our Portland offices in John's Landing.

With hard work comes many rewards, including but not limited to:

• Comprehensive medical, vision and dental insurance
• Flex-time
• Parking / Commute expenses covered
• Fault tolerant whiteboards (n+1)
• Organizational performance based bonuses

Please send messages directly to myself [thomas] @ {networkredux}.com.

Network Redux Introduces Redux Referral

(July 20, 2011, Posted in General News )

We want to thank you, our awesome clients, for helping spread the word about Network Redux.

Starting immediately, we will send you a $10 gift card for every friend you refer that signs up for a free, no obligation Network Redux Dashboard account. If your friend purchases a Network Redux VS or EVS, we’ll generously reward you with cash or service credits.

How to participate in the Network Redux Referral Program:

1. Sign into your [Network Redux Dashboard] (https://dashboard.networkredux.com)
2. Click the “Referral Program” tab on the left side of your dashboard screen.
3. Send your personalized referral link (located on the top right of the screen) to all of your friends. (Note: Your friend must visit our site using your personalized referral link to get credit.)
4. Receive and spend your $10 gift card to Amazon.

It’s that easy. Really.

Start telling your friends about Network Redux [right now] (https://dashboard.networkredux.com)!

Do you have a Social Media Company Guide?

(July 18, 2011, Posted in General News )

Over the past few weeks, I’ve been preparing a written marketing strategy document for Network Redux. I was trained in Public Relations and Journalism at the University of Wisconsin-Madison and spent the majority of my career in a marketing and public relations firm. Thus, I have a very systematic approach when implementing long-term marketing strategies.

Right now I am working on the social media portion of the plan and developing a social media guide for Network Redux. If you don’t have a formal Social Media Company Guide and you have more than two employees working at your company, I suggest you spend some time thinking about official guidelines to Tweet/post by.

Below are a few highlights from our Social Media Company Guide. I’m happy to share the full document as a guide as you create your own. Email me if you are interested.

Keep logins and passwords in one place. Be sure you know where all of your company’s social media account logins and passwords are located. We recently made the rookie mistake of losing the login and password for our Facebook account and had to change our Facebook vanity URL. Keep the passwords and logins close to the Social Media Company Guide to prevent personnel changes from affecting your social networking presence.

Be genuine. Stick to posting and commenting on subjects that are in your area of expertise. You won’t catch me tweeting or blogging on the Anatomy of High Availability or about the technologies behind the Network Redux Dashboard. Luckily I can rely on an amazing team to post some real content rich truffles. Tap into your client base or community if want to blog about something you’re not an expert at. Guest bloggers resonate more with an audience than wanna-be bloggers.

Think before your post. Remember that most social media platforms are just a large, searchable and public message board. Do not post anything that you would not want your largest client, your dream client or your mom to read. I am constantly reminding myself that Twitter is donating its entire archive of public tweets to the Library of Congress.

Criticism is inevitable. Everyone is a critic in the social media realm. Be prepared for someone to say something negative about you or your business. When this happens, DO NOT ENGAGE. Acknowledge the issue publicly and then, if necessary, take conversation with the naysayer off the social media platform. Any sort of public confrontation with a member of your community is unprofessional even if it is behind a computer screen.

What spoken, or unspoken, social media guidelines do you follow? Do you have a formal Social Media Company Guide? If you don’t think one is necessary, why not?