I don't entirely understand the calculation for starting_gas_height and finishing_gas_height, what's the point of that above just computing startgas*len(tx_set) for each tx_set in rangeset R?

interesting. Can you give an example of when this would be best taken advantage of?

Wondering why this is at transaction level, not contract level. If a contract could specify what other contracts it can communicate with, then all transactions sent to it would be computed in parallel, and as a bonus the blockchain itself could eventually be.

Given addresses are randomly distributed throughout the address space, I'm not sure how useful a prefix-based scheme is likely to be - how often will contracts you wish to interact with be conveniently clustered under a common prefix?

@MaiaVictor What if the contract can communicate with other contracts specified at runtime?

Once we have #98 clients can also implement optimistic concurrent processing of txs. The assumption here would be, that most txs in a block don't share any state, except for block.coinbase. So we'd need a different mechanism to collect fees for the miner, but otherwise this can be done without any further protocol updates (a hint on parallelizable group of txs in the transaction list, might be helpful though).

@Arachnid From my understanding, you could maybe have 10 contracts for a given Token (e.g. SNT) where each SNT contract only has access to 1/10 of the address space, with the ability to do cross-token-contract swaps as well. If this is possible, then you could make SNT transactions parrelizable to some extent.

That's the best example I could come up with, but I am sure there are more sensible ones.

I would rather that DELEGATECALL (and for the sake of symmetry, CALLCODE) worked out-of-range. Blindly deploying a contract with libraries is unlikely to group them in the same range. Technically, one could brute-force them into doing so by grinding away at addresses and nonces, but this would be incredibly user-unfriendly. It is also backwards-incompatible.

I do realize that a DELEGATECALL could fail if the targeted contract selfdestructed. One possible solution would be that a DELEGATECALL uses the code of a contract at the beginning of a block, and ignore selfdestructs for this one purpose. That adds a wart, but it also improves the life of the average user.

All that said, I agree with @Arachnid. In a way, this penalizes non-address-grinded multi-contract dapps, For that matter, if address grinding ever became a significant factor in dapp design, users will inevitably want to cluster themselves around popular ranges, leading to pseudo-shards. Elegant and emergent in its own bizarre way, but I don't think it would be desirable compared to an actual first-class sharding system.

@heikoheiko: I'm curious why sharing block.coinbase is a problem. Unless that one address is involved with the transaction, there should be no difference whenever it receives rewards.

Once we have #98 clients can also implement optimistic concurrent processing of txs. The assumption here would be, that most txs in a block don't share any state, except for block.coinbase. So we'd need a different mechanism to collect fees for the miner, but otherwise this can be done without any further protocol updates (a hint on parallelizable group of txs in the transaction list, might be helpful though).

Agree. However, there is currently no gas incentive to make nicely parallelizable txs, and furthermore the worst case still has zero parallelization, and client devs would still have to design for the worst case. This EIP increases the gas limit for "normal usage" by up to 8x but does NOT worsen the worst case at all, and it also introduces an up to 8x subsidy for parallelizable txs.

Given addresses are randomly distributed throughout the address space, I'm not sure how useful a prefix-based scheme is likely to be - how often will contracts you wish to interact with be conveniently clustered under a common prefix?

Most transactions only need to interact with one contract; with EIP86 maybe two. To give one trivial example, any transactions that use two distinct ERC20 tokens will usually have independent read/write ranges.

If a contract could specify what other contracts it can communicate with, then all transactions sent to it would be computed in parallel, and as a bonus the blockchain itself could eventually be.

Consider things this way:

1. (a) There exist contracts that only need to interact with a few other contracts. (b) There also exist contracts that could theoretically interact with anything.
2. (a) There exist transactions that only need to interact with a few known contracts. (b) There also exist transactions that could theoretically interact with any address.

Ideally, we want to retain support for 1b and 2b, but recognize and subsidize 1a and 2a. If we recognize 1a in protocol, then we don't recognize all of 2a, because there are cases where a contract could theoretically interact with anything (quick example: decentralized exchange), but the transaction can be more static (one particular order in the DEX with a known counterparty). However, any instance of 1a is also an instance of 2a, as if a given contract C only affects D1, D2 and D3 then any transaction sending to C can only affect C, D1, 2 and D3. Hence, 1a is a subset of 2a, and so 2a is the more general thing to recognize and subsidize in protocol.

interesting. Can you give an example of when this would be best taken advantage of?

Some quick examples:

1. Transactions going to an ICO can happen in parallel to almost everything else, so the ICO would not interfere with normal traffic the same way. In fact, Status-style network congestion would not be possible unless there were 8 ICOs happening at the same time.
2. ERC20 tokens can be parallelized (ie. transactions sending MKR and transactions sending REP can be totally separated)

I would even go so far as to say that nearly all activity on the current ETH chain would be parallelizable.

I would rather that DELEGATECALL (and for the sake of symmetry, CALLCODE) worked out-of-range.

This is what specification v2 is for. Libraries are generally read-only, and v2 allows for intersecting read ranges, so it should cover all of these use cases.

how often will contracts you wish to interact with be conveniently clustered under a common prefix?

I think you might have misunderstood. My scheme does not require you to choose a single prefix. You can set the read or write range for a transaction as the union of multiple prefixes. So address grinding should not be necessary or particularly helpful.

I don't entirely understand the calculation for starting_gas_height and finishing_gas_height, what's the point of that above just computing startgas*len(tx_set) for each tx_set in rangeset R?

Not sure I understand your proposal. The intuition behind my proposal is that the finishing_gas_height of a transaction can be thought of as an upper bound on the computational "depth" of the transcation - that is, the number of clock cycles of time needed to execute that transaction and all of its dependencies that can't be parallelized.

I'll give an example. Suppose you have four transactions, and suppose that the full address range is the letters ABCDEFGHIJ for simplicity. The first tx has 25000 gas and read/write ranges ABCD. The second tx has 50000 gas and read ranges EFGHIJ and write ranges EFG. The third tx has 65000 gas and read ranges EFGHIJ and write ranges HIJ. All three of those txs can be executed in parallel. Now, suppose the fourth tx has 20000 gas and has read/write ranges CDEF. Executing this tx requires having already executed the first and the second, though not the third (as the third does not write anything that the fourth reads or writes); hence its finishing_gas_height is max(25000, 50000) + 20000 = 70000. A computer with 8 threads would be able to finish processing all four transactions in 70000 cycles (assuming 1 cycle = 1 gas); hence these 4 transactions could fit into a block with a 70000 gas limit.

Hope that makes some sense.

I like this proposal a lot. Given that developers usually opt towards "pull" rather than "push" transactions, most transactions should be able to take advantage of this.

If the subsidy is removed, could this be implemented as a soft fork? Client devs may struggle with multi-threaded implementations and I don't see a pressing need for non-miners to implement this.

@vbuterin I have a question on finishing_gas_height which you described as the computational depth of the transaction.

In your example 4 transactions you calculate: finishing_gas_height(T3) = max(25000, 50000) + 20000 = 70000

The number 70000 is the maximum gas needed to execute that transaction and all of its dependencies that can't be parallelized. Correct?

But then you mention that:

these 4 transactions could fit into a block with a 70000 gas limit

Unless I am missing something, the block's max gas limit required shouldn't change due to the way you process the transactions .Shouldn't it be 160000, basically the sum of all transactions of the block?

Some additional comments on the actual spec:

I really like the idea and believe it will really help relieve the network pressure in Status-like ICO situations.

Even though I suppose it would be a bit more work, I think that spec V2 where you specifically specify both read and write ranges of the transaction is far superior since it will allow for more efficient parallelization.

Unless I am missing something, the block's max gas limit required shouldn't change due to the way you process the transactions .Shouldn't it be 160000, basically the sum of all transactions of the block?

This EIP will change the role of the block gas limit. So a 4.7m gas limit would be not just 4.7m gas worth of computation, but rather 4.7m gas worth of time on an 8-core machine. Perhaps renaming it to gas_height_limit would make that clearer?

Even though I suppose it would be a bit more work, I think that spec V2 where you specifically specify both read and write ranges of the transaction is far superior since it will allow for more efficient parallelization.

Agree. I definitely believe v2 is superior.

One thing that should be noted here is that the CPU-intensive workload should scale linearly with the number of cores, but what does not scale that well is the size of memory and database access. It could be that this change will effectively make state reads 8 times cheaper in gas, depending on the database.

On the other hand, if the transaction gives an explicit list of addresses (i.e. not only prefixes) it will read from, we might already load these accounts into the memory of the core processing this transaction before we start processing it.

Extremely in favor of this proposal. In addition to the immediate scaling gains, I have a strong hunch that defining and incentivizing parallelism will open up very useful design space for us in the future.

If we do this, let's do v2 if at all possible. A lot of current design patterns heavily use delegatecall for efficiency reasons, and it would be unfortunate to disincentivize that.

but what does not scale that well is the size of memory and database access. It could be that this change will effectively make state reads 8 times cheaper, depending on the database.

Good point. Question: are SSD reads parallelizable? If not, then state reading opcodes would definitely need to be increased in price.

we might already load these accounts into the memory of the core processing this transaction before we start processing it.

One problem with this is that accounts can have arbitrarily large storage trees, and we can't load that into memory.

@vbuterin

So a 4.7m gas limit would be not just 4.7m gas worth of computation, but rather 4.7m gas worth of time on an 8-core machine.

I see. That makes sense.

Perhaps renaming it to gas_height_limit would make that clearer?

Yes I think this sounds like a bit more accurate description for this field. So a block would have a gas_height_limit and the gas_used in the block can still be greater than that with a maximum theoretical limit (in the case of full parallelization) of 8 * gas_height_limit.

Ideally, we want to retain support for 1b and 2b, but recognize and subsidize 1a and 2a. If we recognize 1a in protocol, then we don't recognize all of 2a, because there are cases where a contract could theoretically interact with anything

Makes sense. I didn't put a lot of thought on this, so that was just speculation, but let me explain what I had in mind: if a contract pre-determined a set of contracts it can interact to, then, in situations of stress, there could (guessing) be an automated way to fork the contract from the rest of network, letting it run on its own cluster for a day or two. It would be a separate, merge-mined network for that time. Eventually, it could be re-merged. Assuming splitting and merging was a costly operation, that would only be possible if we had a guarantee the contract won't receive a transaction that interacted with contracts on the other network. That is why the restriction couldn't be on transaction level, as any transaction could trigger a re-merge.

@MaiaVictor What if the contract can communicate with other contracts specified at runtime?

Then it wouldn't be parallelizable, I guess. Except if we had add_friend and remove_friend opcodes to modify the set of contracts a contract can interact with? Both would take a few days to concretize, in order to give the networks time to split/merge, under my speculative and possibly wrong dynamic-forking scheme.

if a contract pre-determined a set of contracts it can interact to, then, in situations of stress, there could (guessing) be an automated way to fork the contract from the rest of network, letting it run on its own cluster for a day or two

Sure. But this is not at all compatible with the goal of "easy parallelizability" :)

Networks splitting and merging is the sort of thing we're considering for later stages of the sharding roadmap, whereas this proposal is intended to be more here-and-now.

@chriseth Given that the state trie is keyed by sha3(address) and not address, although this proposal allows good parallelization of compute-intensive transactions, it likely won't lead to a reduction in disk reads as trie lookups will still have to traverse completely from the root and state lookups won't be any cheaper.

Only lookups of fully pre-determined addresses would benefit, although we'd have to charge for "pre-loading" gas ahead of time in case the transaction never actually inspects those accounts.

@vbuterin

A computer with 8 threads would be able to finish processing all four transactions in 70000 cycles (assuming 1 cycle = 1 gas); hence these 4 transactions could fit into a block with a 70000 gas limit.

This assumes that disk/memory operations are parallelizable, when in fact we usually find these to be some of the largest bottlenecks while optimizing.

Can we cement this with a simple example?

Suppose I call a contract with 150,000 gas in thread A, which moves tokens in thread B (utilizing 50,000 gas). Given that there are 8 threads, how much gas does this transaction consume?

Edit: I think I'm confused about the gas consumption ramifications. Is this proposing individual transactions spend only some fraction of gas commensurate with the subset of the state space they update? Or is it proposing a way to pack more gas spending into a block with a fixed gas limit?

Question: What are the tools that need to be made available for handling interdependent state changes of contracts within the read and writing range?

Or is this proposal simply proposing to handle only the independent ones?

Is this proposing individual transactions spend only some fraction of gas commensurate with the subset of the state space they update? Or is it proposing a way to pack more gas spending into a block with a fixed gas limit?

The latter.

This assumes that disk/memory operations are parallelizable, when in fact we usually find these to be some of the largest bottlenecks while optimizing.

Something can be a bottleneck and be parallelizable at the same time. Currently, the problem is that each individual database op may be dependent on the result of the previous, whereas with this scheme you could be fetching several storage keys at the same time.

I'd definitely be very interested in getting more data on parallelization and SSD reads.

TL;DR: Is it a known fact that the bottleneck is transaction processing, and not network latency (or something else altogether)?

As much as I like the proposal in general, got to highlight: it does not immediately follow that introducing parallelisation like this will result in a gas_limit increase (as in its current meaning).

Imagine a newly-received block being validated by a node. Say for example that:

• 1/4 of the txs are "batches of MAX_THREADS", which can be processed in parallel;
• 1/4 are "batches of [2; MAX_THREADS)";
• 1/4 are new-style "batches of 1" - calls to contracts with a CREATE operation, and therefore accesses to an address that can only be determined at execution time;
• 1/4 are "old-style unbatched".

A miner pool's node is likely to be capable of many more concurrent threads than the largest batch here, yet even they have to process the latter two categories one tx at a time. Slower miners that can't provide such concurrency are likely to vote the gas limit down, if their uncle probability increases.

Many more miners may come to be seen as "slow" if the bottleneck turns out to be not validation time, but block propagation time (i.e. time in transit). This parameter is likely to increase, since for the same new gas_height_limit as the old gas_limit, a block may be up to MAX_THREADS bigger byte-wise.

Sure, slow miners would eventually be forced off the market by their own inefficiency. But that may take a while - at least 'till unprofitability of such behaviour becomes apparent to their operators. The time period, I'd make a guess, is months, if not years.

IMO it's still worth diverting mindshare towards this goal from other development (probably because it's not my own mindshare being diverted q:D).

Just something to keep in mind - this proposal is about activating processing power that's currently underused, to put market pressure on those that are grossly under-performing. It may back-fire, though, by using an unexpected criteria to determine who it is that's "under-performing". If that's the criteria turns out to be network latency, then the answer may be "everybody".

Also, if implemented, this will likely introduce "fee levels" for various categories of transactions, where a "batch of 1" (or "old-style unbatched") would need to have gas price at least MAX_THREADS higher than the most efficiently parallelisable one.

Not calling this part "bad", only that such an outcome would be a rational market response.

calls to contracts with a CREATE operation, and therefore accesses to an address that can only be determined at execution time

This part is not necessarily true; contract creation is in many cases deterministic. Though in some portion of cases (eg. where each operation in some dapp causes the same contract to create a new contract, with an incrementing nonce determining the address) it's certainly difficult, and in those cases, yes, you would have to just pay the MAX_THREADS times higher gas prices.

TL;DR: Is it a known fact that the bottleneck is transaction processing, and not network latency (or something else altogether)?

Network bandwidth can be fairly easily scaled up; just get a better internet connection. IO can also be scaled up; if all else fails, require nodes to have 512 GB of RAM. Transaction processing, however, is single-threaded, which makes it especially hard to scale at this point. This proposal fixes that side of things; I agree that optimizations on the other two are also necessary.

@vbuterin SSD reads are, in principle, able to be satisfied in parallel. However, there are a bunch of gotchas:

• Most consumer SSDs still use SATA, which, even with NCQ, only practically supports a queue depth of 31. Newer standards like NVMe support many, many more simultaneous drive reads and writes.
• SATA devices are single-queue, so there will be lock contention in the kernel to queue drive reads. Additionally, some drive commands cannot be queued while there are other commands in the queue.
• Even if multiple reads can be issued to the drive simultaneously, read bandwidth may be constrained by the number of simultaneous ongoing DMAs.
• This is firmware-dependent, but you probably can't read from a block being erased, so reads within the same erase block as a pending write will likely get stalled. (Erase blocks tend to be on the order of 128kB.) Keep in mind that userspace programs don't typically get to control the on-disk block layout of their files, so this isn't something that is necessarily easy to mitigate directly.
• Some SSDs are much smarter than others in terms of their queue management. It requires some cleverness on the part of the hard drive to re-order writes with respect to reads, and consumer hard drives are far from clever. (If you've ever been curious why Intel can charge $3000 for an enterprise SSD, this is why.)
• POSIX file-range locking semantics can create locking bottlenecks in the kernel in poorly-designed applications.

TL;DR implementation quality and hardware/firmware quality make an enormous difference.

Source: filesystems work used to be my full-time job.

Edit: there's a lot of implementation improvement that could be made on the geth and parity storage implementations, given the specific constraints of what the EVM can actually store (fixed-size keys and values, and a huge advantage for having a log-structured COW backing, since you need a kind of snapshotting mechanism to do rollbacks of aborted transactions.)

Edit 2: SSTORE and SLOAD operations are already very under-priced relative to other operations. Consider that ADD can be carefully implemented to run in 3ns, while the read latency on an expensive NVMe flash drive is 20,000ns (and about double that when you include software latency on a well-provisioned server.) Since ADD is 3 gas and SLOAD is 200, we'd expect the average storage read to be satisfied in an (amortized) time of 200ns, which isn't even possible in the best case.

SSTORE and SLOAD operations are already very under-priced relative to other operations. Consider that ADD can be carefully implemented to run in 100ns, while the read latency on an expensive NVMe flash drive is 20,000ns (and about double that when you include software latency on a well-provisioned server.) Since ADD is 3 gas and SLOAD is 200, we'd expect the average storage read to be satisfied in an (amortized) time of 6,667ns, which seems unrealistic.

Note we can't rely on averages or amortisation - as long as an attacker can force a worst-case, they will.

I have a nice title for this eip: Vertical Sharding :)

My usual preferred lingo is that parallelization is something that can happen within nodes, whereas sharding specifically is between nodes (and particularly, between different administrative domains). Though I'll admit this is not settled terminology :)

I agree with that terminology.

Sharding is internode/interdomain, and I guess its name derives from shards of the Ethereum crystal, and is a subcomponent or channel of the network.
Parallelization is intranode, and is related to parallel threads in a computer.
intra = within
inter = between

Referenced here:
https://ethresear.ch/t/the-stateless-client-concept/172

"A solution is to require a transaction to include a static list of the set of accounts that it can access; like EIP 6488 but much stricter in that it requires a precise enumeration rather than a range."

"The requirement for transactions to specify account lists incidentally adds a high degree of parallelizability; this is in many ways a supercharged version of EIP 648."

Will this create an incentive for miners to include transactions that have a small address coverage range before ones that have a larger range, since they could stack more of the first kind? Asking because this could influence how projects build their multi-contracts infrastructure.

@PhABC I've pondered this in an earlier comment. The gist - it'd still primarily be determined by gas*gasPrice provided with the tx.

That is, there IMO would be no economic incentive to include N fully-parallelisable transactions with G in fees each, as compared to one unparallelisable transaction with N*G in fees.

To be clear, the only changes needed by transaction originators/wallets would be to add ranges when we build transactions?

Reference implementations of RLP generation would also be great so we can design client-side JS libraries.

Considering Cryptokitties...this ought to be moved up the priority queue.

The only direct economic incentive seems for miners: with this proposal they can fit MAX_THREADS more transactions in a block and earn more fees.

What about an alternative approach where there are no new transaction types, but transactions in a block are grouped in batches of independent transactions? The client would optimistically execute all the transactions in each batch and reject the block if there is a non-empyty intersection; each batch would have an effective batch_start_gas = max(start_gas) * batch_size / MAX_THREADS and batch_finishing_gas = max(gas_used) * batch_size / MAX_THREADS.

This solution does not require additional range checks while the EVM is running on the client node (these checks can be done after each batch has been executed) and can immediately give an effective discount to old-style transactions, if the miner is smartly batching them.

As discussed at the sharding workshop last week, above definitions of starting and finishing gas heights can lead to suboptimal results if transactions are not ordered properly (or I have misunderstood the proposal). Example:

• MAX_THREADS = 2
• gas_limit = 150k
• tx1.gas_used = 50k
• tx2.gas_used = 50k
• tx3.gas_used = 100k
• tx4.gas_used = 100k
• All access lists are non-overlapping

If the transactions are ordered [tx1, tx2, tx3, tx4] the "gas ranges" (starting_gas_height and finishing_gas_height are as follows:

• tx1: 0 - 50k
• tx2: 0 - 50k
• tx3: 50k - 150k
• tx4: 50k - 150k

However, if the order is [tx3, tx1, tx2, tx4]:

• tx3: 0 - 100k
• tx1: 0 - 50k
• tx2: 100k - 150k
• tx4: 100k - 200k

So in this order the global gas limit would be exceeded, although a sensible miner strategy can execute the transactions in the same amount of time ("schedule the next transaction if there's a free thread and it doesn't overlap with any non-executed earlier transaction").

I think this could be fixed by keeping track of MAX_THREAD separate gas heights and, for each transaction, choosing the lowest possible one under the constraint of no conflicting parallel transactions.

I wanted to point this out, but in my opinion it is not worth implementing this more complicated algorithm. Both methods appear be equivalent if the miner (or, more generally, the proposer) chooses an optimal transaction order in the first place. This needs to be done anyway in order to maximize gas usage if transactions do have overlapping access lists.

Whatever happened to this? Long ago it was slated for metropolis

@joeykrug, cross-posting my reply from reddit:

It was discussed on an AllCoreDevs call around the time (it was never "slated for Metropolis", as in it was never officially Accepted for inclusion in the next HF on an All Core Devs call). As I recall, the feedback was that increasing the number of CPU threads doesn't address the actual bottleneck in scaling tx throughput, which is disk I/O (not CPU processing). You can see this feedback in the github issue itself and in vbuterin's reply:

This assumes that disk/memory operations are parallelizable, when in fact we usually find these to be some of the largest bottlenecks while optimizing.

I'd definitely be very interested in getting more data on parallelization and SSD reads.

Once it became widely realized that disk I/O is the bottleneck, research began on optimizing the state trie's data structure, and how it might be possible to parallelize SSD reads: Roadmap for Turbo-Geth.

After EIP 648, the next research thrust was on stateless clients: "this is in many ways a supercharged version of EIP 648." Stateless clients would enable a sort of 'parallelized state i/o' because all accessed state is pre-specified in the witness list (i.e. all accessed state is pre-loaded from disk into memory before the transaction is processed; in theory disk I/O of the validating node is no longer the bottleneck).

In subsequent months, discussions around sharding and stateless clients mainly revolve around the question of "stateless clients vs storage rent". And that brings us to the present day, ime.

Would this not allow the DAO-style soft forks, and with it, censorship? http://hackingdistributed.com/2016/07/05/eth-is-more-resilient-to-censorship/

If the ranges are tight, then miners can cheaply figure out what are the effects of the transaction, and censor them. If the range are broad, then there is less effect of the parallelisation. They way I see it - it is trading off censorship resistance for more parallel execution. Not sure this tradeoff is worth doing

There has been no activity on this issue for two months. It will be closed in a week if no further activity occurs. If you would like to move this EIP forward, please respond to any outstanding feedback or add a comment indicating that you have addressed all required feedback and are ready for a review.

This issue was closed due to inactivity. If you are still pursuing it, feel free to reopen it and respond to any feedback or request a review in a comment.

Why not use a similar approach to BlockSTM? I think Polygon has already implemented it for EVM or at least an MVP.