Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pcntl_fork causing “errno=32 Broken pipe” #474

Closed
penfold45 opened this issue May 17, 2014 · 45 comments
Closed

pcntl_fork causing “errno=32 Broken pipe” #474

penfold45 opened this issue May 17, 2014 · 45 comments

Comments

@penfold45
Copy link

Hi

I am having issues with PHP notices after upgrading from php 5.4.26 to 5.4.28. I original thought it was a php issue but it may be related to this Redis extension.

The full details can be found here http://stackoverflow.com/questions/23713480/after-php-upgrade-pcntl-fork-causing-errno-32-broken-pipe

But basically when I am forking, whenever a child closes, except for the first child, I get an error saying:

Notice: Unknown: send of 6 bytes failed with errno=32 Broken pipe in Unknown on line 0

It feels like the first child is closing the connection to Redis and then the following children try and close the same connection.

It appears that this has only just started being an issue since either PHP 5.4.27 or 5.4.28

Could this be a phpredis issue?

I have Redis Version => 2.2.5 installed

@michael-grunder
Copy link
Member

Hey there,

While I don't know specifically why, I have a strong suspicion that the Zend library might get a little quirky around sockets within the context of a forking. That being said, I'll see if I can replicate your issue based on the stack overflow question you mentioned.

Cheers dude,
Mike

@penfold45
Copy link
Author

Brilliant thanks. For the time being I have just disabled php notice errors for the scripts that fork. Makes my log files a little quieter ;)

@penfold45
Copy link
Author

Hi. I was woundering if you have had chance to look into this yet?

@penfold45
Copy link
Author

Bump*

@jessem
Copy link

jessem commented Oct 29, 2014

+1 Also seeing this issue with workers that fork in PHPResque https://github.com/chrisboulton/php-resque

@splitice
Copy link

+1 also seeing this in php resque workers.

Appears harmless... guessing its "QUIT\r\n"

dominics pushed a commit to vend/php-resque that referenced this issue Nov 18, 2014
@jat001
Copy link

jat001 commented Jan 12, 2015

I know how to fix this bug. Just close the connect to redis before pcntl_fork and re-connect to redis after pcntl_fork.

@anton000
Copy link

no updates?

@roeyb
Copy link

roeyb commented Feb 3, 2015

+1

3 similar comments
@metal-young
Copy link

+1

@rlweb
Copy link

rlweb commented Feb 20, 2015

+1

@schlaus
Copy link

schlaus commented Mar 20, 2015

+1

@lexaurin
Copy link

lexaurin commented Mar 26, 2015

Same here. PHP 5.5.9, phpredis 2.2.7

Simple script to replicate this bug https://gist.github.com/lexaurin/8897f1be48b9e8747b7d/e8a5fcd750623b6fc250f9c529fc2e65cc55893a

Expected result:

bool(true)

Actual result:

bool(true)
PHP Notice:  Unknown: send of 6 bytes failed with errno=32 Broken pipe in Unknown on line 0

Notice: Unknown: send of 6 bytes failed with errno=32 Broken pipe in Unknown on line 0

PHP Notice:  Unknown: send of 6 bytes failed with errno=32 Broken pipe in Unknown on line 0

Notice: Unknown: send of 6 bytes failed with errno=32 Broken pipe in Unknown on line 0

Connections before script (lsof -i :6379):

COMMAND     PID USER   FD   TYPE   DEVICE SIZE/OFF NODE NAME
redis-ser 20319 root    4u  IPv4 10422993      0t0  TCP *:6379 (LISTEN)

Connections after first child forked:

COMMAND     PID USER   FD   TYPE   DEVICE SIZE/OFF NODE NAME
redis-ser 20319 root    4u  IPv4 10422993      0t0  TCP *:6379 (LISTEN)
redis-ser 20319 root    6u  IPv4 10446216      0t0  TCP config0.local:6379->config0.local:52050 (ESTABLISHED)
php       21993  lex    8u  IPv4 10446860      0t0  TCP config0.local:52050->config0.local:6379 (ESTABLISHED)
php       21994  lex    8u  IPv4 10446860      0t0  TCP config0.local:52050->config0.local:6379 (ESTABLISHED)

Connections after second child forked:

COMMAND     PID USER   FD   TYPE   DEVICE SIZE/OFF NODE NAME
redis-ser 20319 root    4u  IPv4 10422993      0t0  TCP *:6379 (LISTEN)
php       21993  lex    8u  IPv4 10446860      0t0  TCP config0.local:52050->config0.local:6379 (CLOSE_WAIT)
php       22020  lex    8u  IPv4 10446860      0t0  TCP config0.local:52050->config0.local:6379 (CLOSE_WAIT)

Connections after third child forked:

COMMAND     PID USER   FD   TYPE   DEVICE SIZE/OFF NODE NAME
redis-ser 20319 root    4u  IPv4 10422993      0t0  TCP *:6379 (LISTEN)

Note: Fact that child closes "both" connections at the end is right (since it shares file descriptor with parent).

@lexaurin
Copy link

lexaurin commented Mar 26, 2015

Modified my gist a bit to see which process produces notice https://gist.github.com/lexaurin/8897f1be48b9e8747b7d/d46ccd5c2ffabb1c3a09118db80654cb382c6c96
and it's interesting - third child at the end and following children and then parent at the end.

My log:

bool(true)
Parent pid: 22968
Child pid: 22969
Child pid: 22971
Child pid: 22972
Error in PID 22972: Unknown: send of 6 bytes failed with errno=32 Broken pipe
Child pid: 22973
Error in PID 22973: Unknown: send of 6 bytes failed with errno=32 Broken pipe
Child pid: 22974
Error in PID 22974: Unknown: send of 6 bytes failed with errno=32 Broken pipe
Error in PID 22968: Unknown: send of 6 bytes failed with errno=32 Broken pipe 

@sanvibyfish
Copy link

I have same problem, I use pconnect to connect redis.

After fork, I pconnect again, But nothing help.

Sometime always throw exception like this.

Exception Redis::lLen(): send of 38 bytes failed with errno=32 Broken pipe in

This code just manager jobs, and run it.

        $files = glob(__DIR__ . '/jobs/*.php');
        foreach ($files as $_file) {
          $this->workers[$name . $i] = pcntl_fork();
          $redis = new Redis();
          $redis->pconnect('127.0.0.1', 0.3);
          if ($this->workers[$name . $i] === 0) {
            posix_kill(posix_getpid(), SIGTERM);
          }
        }
        while (count($this->workers)) {
            $pid = pcntl_wait($status);
            usleep(50000);
        }

Then the unix crontab will run it. Same job will pconnect Redis, after pcntl_fork() pconnect Redis can't fix

@lixiphp
Copy link

lixiphp commented Oct 19, 2015

+1

@driq
Copy link

driq commented Nov 2, 2015

Having the same issue when using https://github.com/kriswallsmith/spork

@lixiphp
Copy link

lixiphp commented Nov 3, 2015

this bug is from php pcntl, just in the worker colse redis, will solve this problem.

@giraclion
Copy link

Same issue happens when using PHP 5.5.9 + phpredis 2.2.7 + redis engine 2.8.19 + Magento CE 1.9.0.1 + Cm_Cache_Backend_Redis environment.
Error:

Redis::exec(): send of 8192 bytes failed with errno=32 Broken pipe

@jaymecd
Copy link

jaymecd commented Mar 14, 2016

+1

3 similar comments
@ricardosierra
Copy link

+1

@piotrkochan
Copy link

+1

@jimyaghi
Copy link

jimyaghi commented Jun 8, 2016

+1

@sapzxc
Copy link

sapzxc commented Jul 9, 2016

+1 :)

@vanquang9387
Copy link

+1

2 similar comments
@consatan
Copy link

+1

@damonto
Copy link

damonto commented Sep 19, 2016

+1

@robbieaverill
Copy link

robbieaverill commented Oct 6, 2016

You guys know there's now a "+1" equivalent "like button" on GitHub right? Without spamming the participants with emails (unless that's your objective!)

@j-d
Copy link

j-d commented Nov 2, 2016

+1

@7c
Copy link

7c commented Nov 18, 2016

php 5.5.9 suddenly started doing same behaviour. I cannot use it anymore in production. This is high priority bug

@yatsukhnenko
Copy link
Member

Could someone provide script which cause this error?

@yatsukhnenko
Copy link
Member

@lexaurin, thanks.
There is no error on my gentoo box

php redis-pipe-bug.php 
bool(true)
Parent pid: 27921
Child pid: 27922
Child pid: 27923
Child pid: 27924
Child pid: 27925
Child pid: 27926

Tested on php5.3-php7.1

@yatsukhnenko
Copy link
Member

@lexaurin depends environment?

@lexaurin
Copy link

I can't reproduce it on my desktop too (PHP 7.0.8-0ubuntu0.16.04.3)

Problem was on Debian machine a while ago and I don't have access there anymore. PHP 5.5.9, phpredis 2.2.7 at that time.

Maybe someone with "+1" can provide more infomration and test it out.

@yatsukhnenko
Copy link
Member

phpredis 2.2.7 is to old. problem may be fixed long time ago :)

@lexaurin
Copy link

I know, it was reported in 2014. If no one can confirm it on newer versions of phpredis I suggest to close this bug with resolution "Upgrade to 3.x"

@ppanphper
Copy link

ppanphper commented Jan 4, 2017

Env: PHP7.1.0+phpredis-3.1.0
use RedisCluster connect redis cluster server.
when the high amount of concurrency will appear when the following error:
[2017-01-04 03:18:24.999684] [Notice] RedisCluster::set(): send of 3182 bytes failed with errno=32 Broken pipe at /home/development/pandy/Swoole_Pool/RedisPool/RedisClient.php[512]
[2017-01-04 03:18:24.999892] [Notice] RedisCluster::set(): send of 3182 bytes failed with errno=32 Broken pipe at /home/development/pandy/Swoole_Pool/RedisPool/RedisClient.php[512]

@zoglun
Copy link

zoglun commented Mar 23, 2017

Same error happen to me under debian8, redis 3.2.7, with php7.0 & Mediawiki 1.28.
PHP Notice: Redis::set(): send of 82 bytes failed with errno=32 Broken pipe in /wiki/includes/libs/redis/RedisConnRef.php on line 96

@ngoctruong1610
Copy link

Same error happens to me under CentOS 6 , Redis 3.2.1 with PHP 7.0 & Magento 2

[2017-03-25 00:00:38] Subscription Logger Publish profile to message queue.CRITICAL: ErrorException: Redis::hGet(): send of 71 bytes failed with errno=32 Broken pipe in /opt/nestle/magento/releases/20170324062419/vendor/colinmollenhour/credis/Client.php:1044
Stack trace:
#0 [internal function]: Composer\Util\ErrorHandler::handle(8, 'Redis::hGet(): ...', '/opt/nestle/mag...', 1044, Array)
#1 /opt/nestle/magento/releases/20170324062419/vendor/colinmollenhour/credis/Client.php(1044): Redis->hGet('zc:k:b22_GLOBAL...', 'd')
#2 /opt/nestle/magento/releases/20170324062419/lib/internal/Cm/Cache/Backend/Redis.php(211): Credis_Client->__call('hget', Array)
#3 /opt/nestle/magento/releases/20170324062419/vendor/magento/zendframework1/library/Zend/Cache/Core.php(306): Cm_Cache_Backend_Redis->load('b22_GLOBAL_PRIM...', false)

@yatsukhnenko
Copy link
Member

@ppanphper, @zoglun, @ngoctruongbc, do you have a script to reproduce this issue?

@virgofx
Copy link

virgofx commented Mar 28, 2017

In my opinion this should be closed as it's a duplicate of #70 -- the broken pipe is the effective result that leads to all sorts of read/write errors.

@yatsukhnenko Script to reproduce is here: #70 (comment)

@yatsukhnenko
Copy link
Member

Duplicate #70

@weixuan19930
Copy link

Same error happens in swoole worker...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests