Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

error 32 broken pipe #224

Closed
shafiul opened this issue Mar 18, 2014 · 6 comments
Closed

error 32 broken pipe #224

shafiul opened this issue Mar 18, 2014 · 6 comments

Comments

@shafiul
Copy link

shafiul commented Mar 18, 2014

Hi,

My app is using PyMySQL for a SQLAlchemy powered application. I'm getting this error quite a lot.

Can you please advise any suggestions?

PyMySQL installed by pip: PyMySQL (0.6.1)

I've been using PyMySQL for a Tornado and Celery app and running in two linux servers, both having the same problems.

Trace:

[2014-03-15 12:45:01,021: CRITICAL/MainProcess] Task scheduling.sms_tasks.schedule_message[38d06f9e-3833-4951-8249-58c3c7d1858b] INTERNAL ERROR: BrokenPipeError(32, 'Broken pipe')
Traceback (most recent call last):
  File "/webapps/phoenix/lib/python3.3/site-packages/celery/app/trace.py", line 251, in trace_task
    I, R, state, retval = on_error(task_request, exc, uuid)
  File "/webapps/phoenix/lib/python3.3/site-packages/celery/app/trace.py", line 201, in on_error
    R = I.handle_error_state(task, eager=eager)
  File "/webapps/phoenix/lib/python3.3/site-packages/celery/app/trace.py", line 85, in handle_error_state
    }[self.state](task, store_errors=store_errors)
  File "/webapps/phoenix/lib/python3.3/site-packages/celery/app/trace.py", line 118, in handle_failure
    req.id, exc, einfo.traceback, request=req,
  File "/webapps/phoenix/lib/python3.3/site-packages/celery/backends/base.py", line 112, in mark_as_failure
    traceback=traceback, request=request)
  File "/webapps/phoenix/lib/python3.3/site-packages/celery/backends/base.py", line 223, in store_result
    request=request, **kwargs)
  File "/webapps/phoenix/lib/python3.3/site-packages/celery/backends/database/__init__.py", line 47, in _inner
    return fun(*args, **kwargs)
  File "/webapps/phoenix/lib/python3.3/site-packages/celery/backends/database/__init__.py", line 99, in _store_result
    task = session.query(Task).filter(Task.task_id == task_id).first()
  File "/webapps/phoenix/lib/python3.3/site-packages/sqlalchemy/orm/query.py", line 2305, in first
    ret = list(self[0:1])
  File "/webapps/phoenix/lib/python3.3/site-packages/sqlalchemy/orm/query.py", line 2172, in __getitem__
    return list(res)
  File "/webapps/phoenix/lib/python3.3/site-packages/sqlalchemy/orm/query.py", line 2376, in __iter__
    return self._execute_and_instances(context)
  File "/webapps/phoenix/lib/python3.3/site-packages/sqlalchemy/orm/query.py", line 2391, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/webapps/phoenix/lib/python3.3/site-packages/sqlalchemy/engine/base.py", line 664, in execute
    return meth(self, multiparams, params)
  File "/webapps/phoenix/lib/python3.3/site-packages/sqlalchemy/sql/elements.py", line 282, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/webapps/phoenix/lib/python3.3/site-packages/sqlalchemy/engine/base.py", line 761, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/webapps/phoenix/lib/python3.3/site-packages/sqlalchemy/engine/base.py", line 874, in _execute_context
    context)
  File "/webapps/phoenix/lib/python3.3/site-packages/sqlalchemy/engine/base.py", line 1026, in _handle_dbapi_exception
    util.reraise(*exc_info)
  File "/webapps/phoenix/lib/python3.3/site-packages/sqlalchemy/util/compat.py", line 168, in reraise
    raise value
  File "/webapps/phoenix/lib/python3.3/site-packages/sqlalchemy/engine/base.py", line 867, in _execute_context
    context)
  File "/webapps/phoenix/lib/python3.3/site-packages/sqlalchemy/engine/default.py", line 388, in do_execute
    cursor.execute(statement, parameters)
  File "/webapps/phoenix/lib/python3.3/site-packages/pymysql/cursors.py", line 102, in execute
    result = self._query(query)
  File "/webapps/phoenix/lib/python3.3/site-packages/pymysql/cursors.py", line 202, in _query
    conn.query(q)
  File "/webapps/phoenix/lib/python3.3/site-packages/pymysql/connections.py", line 733, in query
    self._execute_command(COM_QUERY, sql)
  File "/webapps/phoenix/lib/python3.3/site-packages/pymysql/connections.py", line 872, in _execute_command
    self._write_bytes(prelude + sql[:chunk_size-1])
  File "/webapps/phoenix/lib/python3.3/site-packages/pymysql/connections.py", line 833, in _write_bytes
    self.socket.sendall(data)
BrokenPipeError: [Errno 32] Broken pipe

I've been using supervisor to run a celery program. Following command is used to start the application:

!/bin/bash

NAME="phoenix_celery" # Name of the application
APPDIR=/webapps/phoenix/src # App directory
SOCKFILE=/webapps/phoenix/run/gunicorn.sock # we will communicte using this unix socket
USER=phoenix # the user to run as
GROUP=webapps # the group to run as
NUM_WORKERS=3 # how many worker processes should Gunicorn spawn

echo "Starting $NAME as whoami"

Activate the virtual environment

cd $APPDIR
source ../bin/activate
export PYTHONPATH=$APPDIR:$PYTHONPATH

Create the run directory if it doesn't exist

RUNDIR=$(dirname $SOCKFILE)
test -d $RUNDIR || mkdir -p $RUNDIR

exec celery worker --app=scheduling -l info
This was referenced Mar 18, 2014
Closed
Merged
@kux
Copy link

kux commented Mar 19, 2014

Broken pipe usually means that the remote peer has abruptly shut down the connection (RST packet).

There might be multiple causes for this:

  • firewalls between you and the server that drop connections
  • server shutting down the connection due to a bad query or various other reasons

If this is not a networking issue then the server might send relevant information before shutting down the connection. PyMySQL doesn't always capture that information.

I had a situation where my queries were too large. The MySQL server was responding me with the proper error code (#1153) and was then shutting down the connection. PyMySQL wasn't capturing the error code. It just raised Broken Pipe. In order to figure out what was actually going on I had to use Wireshark and look at the actual packets. Wireshark has a nice MySQL dissector which makes it very easy to sniff the packets.

@methane methane mentioned this issue Mar 20, 2014
Closed
@shafiul
Copy link
Author

shafiul commented Mar 20, 2014

I switched to mysql connector but the situation did not improve, this time I was getting:

(OperationalError) MySQL Connection not available

So I guess this is not a pymysql issue and further searching gave me a hint that this might be a SQLAlchemy issue for MySQL databases. I've followed the "optimistic" workaround as described here: http://docs.sqlalchemy.org/en/rel_0_9/core/pooling.html#dealing-with-disconnects

I'm not still sure if it's a pymysql issue, I'll update if the workaround works.

@methane
Copy link
Member

methane commented Feb 6, 2015

@shaflul To avoid MySQL's wait_timeout, you can use pool_recycle option of create_engine().
I usually use 10, 30 or 60 for it.

@methane methane closed this as completed Feb 6, 2015
@amerhwitat
Copy link

I have the same problem when using openstack 14 it's not starting instance because of hosts can not be found error and I found this error causes the problem in nova-api.log

2019-01-26 05:01:02.776 17087 ERROR oslo_db.sqlalchemy.engines self._write_bytes(packet)
2019-01-26 05:01:02.776 17087 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1044, in _write_bytes
2019-01-26 05:01:02.776 17087 ERROR oslo_db.sqlalchemy.engines "MySQL server has gone away (%r)" % (e,))
2019-01-26 05:01:02.776 17087 ERROR oslo_db.sqlalchemy.engines DBConnectionError: (pymysql.err.OperationalError) (2006, "MySQL server has gone away (error(32, 'Broken pipe'))") [SQL: u'SELECT 1'] (Background on this error at: http://sqlalche.me/e/e3q8)
2019-01-26 05:01:02.776 17087 ERROR oslo_db.sqlalchemy.engines

nova-api.log

@amerhwitat
Copy link

mysql --max_allowed_packet=32G

!/usr/bin/env python2.7
import time
import mysql.connector

would solve it ...

@methane
Copy link
Member

methane commented Jan 27, 2019

This error is not a bug. This error is caused by various program / configuration / environment issues.

Don't use this issue tracker to discuss about this error. It bother me. Use Stackoverflow or other community to avoid bother OSS maintainers.

@PyMySQL PyMySQL locked as off-topic and limited conversation to collaborators Jan 27, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@methane @shafiul @kux @amerhwitat