Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

No Server Available - Remote to twemproxy connected to EC2 to Elasticache #454

Closed
rcvirtjoule opened this issue May 14, 2014 · 5 comments
Closed

No Server Available - Remote to twemproxy connected to EC2 to Elasticache #454

rcvirtjoule opened this issue May 14, 2014 · 5 comments

Comments

@rcvirtjoule
Copy link

Using Dalli 2.6.3 -
I want to use Ruby Dalli gem w/ twemproxy running on an Ubuntu EC2 machine connected to Elasticache. A remote client connect via telnet -> twemproxy -> Elasticache works fine. Here's a sample that demonstrates my local machine can set and get a key pair in Elasticache through twemproxy. So I know basic comm/firewall is set up right.

$ telnet mytwemproxyipaddress 22123
Trying mytwemproxyipaddress...
Connected to ec2-myec2staticipaddress.compute-1.amazonaws.com.
Escape character is '^]'.
add stuff 0 900 12
test_content
STORED
get stuff
VALUE stuff 0 12
test_content
END

From irb, I can't show Dalli successfully connecting with twemproxy.

$ irb
2.0.0p195 :001 > require 'rubygems'
=> false
2.0.0p195 :002 > require 'dalli'
=> true
2.0.0p195 :003 > dc = Dalli::Client.new("ec2-myec2staticipaddress.compute-1.amazonaws.com:22123")
=> #<Dalli::Client:0x007fdf840727e8 @servers=["ec2-myec2staticipaddress.compute-1.amazonaws.com:22123"], @options={}, @Ring=nil>
2.0.0p195 :004 > dc.set('stuff', 'test_content')
I, [2014-05-14T15:44:59.552786 #41821] INFO -- : ec2-myec2staticipaddress.compute-1.amazonaws.com:22123 failed (count: 0)
Dalli::RingError: No server available
from /Users/mymachine/.rvm/gems/ruby-2.0.0-p195/gems/dalli-2.6.3/lib/dalli/ring.rb:45:in server_for_key' from /Users/mymachine/.rvm/gems/ruby-2.0.0-p195/gems/dalli-2.6.3/lib/dalli/client.rb:328:inperform'
from /Users/mymachine/.rvm/gems/ruby-2.0.0-p195/gems/dalli-2.6.3/lib/dalli/client.rb:183:in set' from (irb):4 from /Users/mymachine/.rvm/rubies/ruby-2.0.0-p195/bin/irb:16:in

'
2.0.0p195 :005 >

I've stepped in a bit to see if I could figure out the problem, but have not been successful. The initial exception is thrown on the @Version = version line (538 in my version) in the connect method of class Server in server.rb. When stepping through KSocket.open, I don't see anything unusual.

def connect
  Dalli.logger.debug { "Dalli::Server#connect #{hostname}:#{port}" }

  begin
    @pid = Process.pid
    @sock = KSocket.open(hostname, port, self, options)

Exception thrown here-----> @Version = version # trigger actual connect
sasl_authentication if need_auth?
up!
rescue Dalli::DalliError # SASL auth failure
raise
rescue SystemCallError, Timeout::Error, EOFError, SocketError
# SocketError = DNS resolution failure
failure!
end
end

Would be great to figure out this issue as Elasticache is a good way to implement a central cache cluster, but there's no way to set up Elasticache to accept remote connections. So a proxy is needed for temporary cases for debugging purposes and twemproxy seems to have what's needed. Getting Dalli to work with twemproxy would be great. Any help is appreciated.
@mperham
Copy link
Collaborator

mperham commented May 14, 2014

twemproxy does not support the memcached binary protocol.

Use an SSH tunnel instead. https://help.ubuntu.com/community/SSH/OpenSSH/PortForwarding#Local_Port_Forwarding

@mperham mperham closed this as completed May 14, 2014
@rcvirtjoule
Copy link
Author

Thanks Mike. After looking at this quite a while I'm stumped on how local port forwarding from my local, non EC2 machine, would help to get to an Elasticache end point. Only my EC2 machines can access Elasticache and I can't port forward directly to Elasticache by definition, as we know. Setting up port forwarding from my local machine to one of my EC2 machines still leaves the request hanging on a port on that machine since there's nothing there to forward it on to Elasticache, something twemproxy can do successfully (above), but not through the binary protocol Dalli uses. Since one can't ssh (at least I couldn't) to Elasticache, I don't know how to set up a local port forward on my EC2 machine to Elasticache, which is what I wanted to do, passing on my local request to Elasticache. I built a quick tcp relay proxy, but it has the same problem as twemproxy due to lack of binary protocol support for Dalli services.

Sorry if I'm being thick, but I don't understand how local port forwarding solves the problem of getting Dalli to talk from an non-EC2 machine to Elasticache either directly or through an EC2 instance I own. Can I ask for more details on how this is accomplished? The basic problem I'm trying to solve is debugging an issue with my app while it's running from EC2 off data from Elasticache. That problem doesn't show up when I run the app locally from a locally loaded cache. I'd like to run my local debuggable app (Rails and ruby processes) using Elasticache data, not local memcached. Network latency, performance, and all the reasons one wouldn't do this in production are noted. I'm sure I can get past this problem, but I'm quite sure the debugging need will come up again in the future.

Any further insights would be appreciated. Thank you.

@mperham
Copy link
Collaborator

mperham commented May 15, 2014

You ssh tunnel thru one of your EC2 boxes. ssh -L22122:elasticachebox:11211 user@myec2box then connect Dalli to
localhost:22122

On Thu, May 15, 2014 at 8:24 AM, rcvirtjoule notifications@github.comwrote:

Thanks Mike. After looking at this quite a while I'm stumped on how local
port forwarding from my local, non EC2 machine, would help to get to an
Elasticache end point. Only my EC2 machines can access Elasticache and I
can't port forward directly to Elasticache by definition, as we know.
Setting up port forwarding from my local machine to one of my EC2 machines
still leaves the request hanging on a port on that machine since there's
nothing there to forward it on to Elasticache, something twemproxy can do
successfully (above), but not through the binary protocol Dalli uses. Since
one can't ssh (at least I couldn't) to Elasticache, I don't know how to set
up a local port forward on my EC2 machine to Elasticache, which is what I
wanted to do, passing on my local request to Elasticache. I built a quick
tcp relay proxy, but it has the same problem as twemproxy due to lack of
binary protocol support for Dalli services.

Sorry if I'm being thick, but I don't understand how local port forwarding
solves the problem of getting Dalli to talk from an non-EC2 machine to
Elasticache either directly or through an EC2 instance I own. Can I ask for
more details on how this is accomplished? The basic problem I'm trying to
solve is debugging an issue with my app while it's running from EC2 off
data from Elasticache. That problem doesn't show up when I run the app
locally from a locally loaded cache. I'd like to run my local debuggable
app (Rails and ruby processes) using Elasticache data, not local memcached.
Network latency, performance, and all the reasons one wouldn't do this in
production are noted. I'm sure I can get past this problem, but I'm quite
sure the debugging need will come up again in the future.

Any further insights would be appreciated. Thank you.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/454#issuecomment-43223984
.

@rcvirtjoule
Copy link
Author

Thank you, got it, that works. I had been unwittingly trying to have Dalli connect to the tunnel via the EC2 host name. Thanks again.

@nagarjunadn
Copy link

Hi I found the solution.

The ElasticCache can be accessble only using the EC2 Instances.
Here I have selected ElasticCache Security Group that I was used for my EC2 Instances-Example:launch-wizard-1. And also I have added the Inbound Rule for the port 11211 of Custom TCP Rule in to launch-wizard-1 Security Group.

Its great. Now I am able to access the ElasticCache from EC2 Instances...!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@mperham @nagarjunadn @rcvirtjoule