Blockchain Update 2: HF20 Progress & Bandwidth Changes

in bandwidth •  12 days ago

In our last blockchain update, we announced that we were making HF20 a priority. Since then, we’ve worked hard to accelerate the release. In today’s post, we will explore an especially important part of HF20: improving the bandwidth formula of the Steem blockchain.

Reforming the Bandwidth Formula

As explained in our previous post:

as we continue to scale the blockchain to more and more users, the bandwidth formula that we use to allocate resource usage across all of those users becomes increasingly important.

Move to Resource Credit System

To address this issue, we have completely rewritten the bandwidth system and created a new, state-of-the-art resource allocation system built around a new unit called “Resource Credits” (RCs).

This resource allocation system is the first of its kind in the blockchain / cryptocurrency space and will be the most advanced “freemium” blockchain model in existence.

RC Design Goals

Three primary design goals guided our development of the new bandwidth system. We wanted to create a system that 1) more efficiently allocates blockchain resources; 2) more accurately measures the true cost of running the blockchain; and 3) enables Steem developers to create more predictable user experiences.

Intelligent Resource Allocation

The most important goal of the resource allocation system is intelligent allocation of the blockchain’s resources. The more efficiently resources are allocated, the more user activity can be sustained. This is a critical component of scaling the blockchain. It is also critical that the system disincentivize the over-consumption of resources by users as well as spam.

Accurate Transaction Cost

Every transaction on the Steem blockchain has an associated “cost” in terms of the strain it places on the blockchain’s resources. For example, storing more data in the blockchain can eventually lead to higher RAM requirements for steemd nodes. Increased resource requirements can lead to higher costs for node operators (including developers, witnesses, and exchanges).

This is why it was critical that the blockchain calculate a transaction’s cost more accurately, based on as many factors as possible, and be empowered to limit which transactions can be performed based on their cost to the network.

These factors could include things such as:

  • Blockchain history size
  • Reindex time
  • State file size
  • Memory usage
  • Disk iops
  • Network bandwidth

Predictable UX

Under the current bandwidth system for Steem, users are given a certain number of “bytes” to use based on the amount of Steem Power (SP) they have. The blockchain reduces or increases the number of bytes they are given based on the activity occurring on the blockchain.

While this solution got us to where we are today, it confuses end-users by providing them with a purely binary choice: either acquire more SP, or be unable to transact on Steem. End-users cannot, for example, be told how much bandwidth they will receive from their purchase of more SP. The amount of bandwidth that a certain amount of SP is worth also changes constantly, depending on the activity level of the network. In other words, as the system stands, it is all but impossible for Steem users to form an accurate mental model of bandwidth.

Enabling the formation of a mental model is critical to satisfying our third design requirement: more predictable user experiences on Steem. The predictable model that will result from this new system will enable simple and effective feedback from user interfaces (UIs) to inform users about their RC usage, their remaining RCs, the cost of new transactions (measured in RCs), and options for purchasing additional RCs by powering up SP if they would like additional bandwidth.

Dynamic Cost Calculation

An important design principle carried over from the previous bandwidth implementation was that of dynamic costs based on the blockchain’s activity, as this is a very efficient system. Transactions should “cost” more during times when the blockchain is busy and less during times when it is not. However, as we’ve discussed above, this approach comes with certain downsides in terms of users being able to form a predictable mental model of bandwidth use.

The solution we propose in our RC system is to have a one-to-one correlation between the number of RCs one can get with a certain amount of SP, but at the same time have the costs of various transactions (measured in RCs) change based on the blockchain’s current activity level. The costs are determined based on an internal, market-based system, which is explained below.

Gamification

An added benefit of the new system is that it will further gamify the Steem experience. When a user can know how many RCs they will have, how long it takes for those RCs to recharge, and how many RCs each transaction will cost, they will be able to make informed decisions about how to spend their RCs.

While a primary objective for Steem is to minimize the cognitive load on users, this system will create new opportunities for developers looking to create more engaging applications. Developers will be able to provide interfaces that encourage their users to think tactically about how best to use their limited resources, because more granular data about resource consumption by individual users will be readily available. In addition, those users who may be inclined to over-consume blockchain resources (i.e. spammers) will be discouraged from doing so, as they will be shown very clearly just how much of their limited resources their efforts are consuming.

Maintaining a Frictionless Experience

It is important to reiterate that the appropriate baseline user experience for Steem is: as little friction and cognitive load as possible. We refuse to offer solutions that interfere with this principle. However, another important aspect of any protocol is that it be maximally flexible. Users who simply want to read a few posts, leave a few comments, and send some Steem should be able to do so with ease. But it is also our responsibility to consider the people who want a more active and gamified user experience, as well as those seeking to waste network resources. In a sense, we want Steem to be a game that everyone can enjoy playing, regardless of the type of user experience they prefer or their motivation for playing.

What are RCs?

The most important thing to understand about RCs is that they will be non-transferable credits given to each account based on how much Steem Power it has, which get “spent” whenever a user transacts with the Steem blockchain. Of course, if RCs did not replenish over time, then eventually everyone would become unable to transact at some point. For that reason, RCs regenerate over time.

Fueling Transactions

Once the RC system goes into effect, all transactions on the Steem blockchain will be given a cost in terms of RCs. If an account doesn’t have sufficient credits, the transaction will not be allowed to occur. This is the same as the existing system, which does not allow transactions to occur if the account does not have sufficient bandwidth.

Resource Budget Pools

The new system differs from the current one in the creation of “resource budget pools” which will be established for each resource type (RAM usage, reindex time, etc.). Each pool could be viewed as the blockchain’s “stockpile” of the corresponding resource. RCs will be the “currency” that the blockchain uses to price the resources in those stockpiles, and the internal medium of exchange for said resources.

Again, this is all happening behind the scenes and will require no conscious engagement by the user. It is simply a technical solution to the problem of maximizing the efficient usage of the computational resources available to the network in a scalable manner. Markets, after all, are arguably humanity’s most practical method of distributing resources among populations beyond the bounds of a small tribe.

Computing Price

The price of each resource will be based on the current level of the stockpile. As the stockpile decreases, the price of that resource (in terms of RCs) increases. In other words, as the stockpile goes down, accounts will have to pay more RCs to use the remaining resources.

Cost in RCs won’t translate into anything like a price increase in terms of STEEM or USD, due to the fact that RCs are non-transferrable. The goal is only to leverage a market system to ration resources, not to create speculative opportunities or manufacture another token that can be used to purchase goods or services.

Computing Cost

For each transaction, the System will statelessly compute a value and exchange rate in RC for resources like CPU megacycles, state memory, and history size.

From the github PR:

If CPU cycles cost 5 RC / megacycle, state memory costs 8 RC / byte, and history size costs 4 RC / byte, a transaction which takes 2 megacycles, creates 50 bytes of state, and has a 150 byte transaction size will cost 25 + 508 + 150*4 = 1010 RC.

Technical Details

For those interested in the technical details behind the new RC Bandwidth system, more information can be found in our wiki article on RC-Bandwidth-Parameters.

Summary

These changes will lead to a far more pleasant, more transparent, and more predictable user experience across all Steem applications, and will more accurately allocate network resources by leveraging an efficient market mechanism.

Discussion Welcome

The RC System is only one aspect of HF20 that we have been hard at work on. Because it is a significant change and a dramatic improvement, we wanted to take this opportunity to thoroughly explore this issue with the community.

If you’d like to learn more about the RC System, or if you have any questions relating to it, please leave them in the comments section below.

Thank you for reading.

Steemit Blockchain Team

Authors get paid when people like you upvote their post.
If you enjoyed what you read here, create your account today and start earning FREE STEEM!
Sort Order:  trending

Overall this looks like a good approach but if you are doing this level of revamp I would take a closer look at how you handle state bytes. Transactions which reduce net state usage should have zero or negative cost for state (negative cost would mean crediting RCs back to the user), and transactions which use state for a longer period of time should cost more than those which use it for a short period of time. Permanent state should perhaps be a separate resource pool of its own and only grow at a very slow rate (making it relatively expensive to use)

State is not really something which can simply grow at a set-rate per block, unless the rate is extremely low (comparable to the rate of hardware technology improvements). Setting a per block rate this low without some allowance for reduction in state usage will probably constrain blockchain utility greatly, and setting the limit too high will open paths for cheap denial of service attacks.

State limit is more of a static parameter (as a function of hardware realities) should perhaps be set by witness vote or community consensus (hard fork). Approaching the state limit should increase the price of adding any state (not per block) and likewise the benefit of reducing state usage.

In any case, this proposed model is certainly an improvement over the status quo which attempts to price everything (including both temporary and permanent state) into bandwidth.

I suggest bundling witness vote decay or expiry in the next HF.

The competition for the top 20 ranks is almost completely stalled.

·

There's no competition. Decay had been discussed for HF16, but I believe it has security risks involved. Either way, it needs some solution, witnesses are doing nothing for Steem right now.

I would also recommend cutting the witness voting to 5 witnesses. There's no reason to vote for 30 witnesses; it'll force people to think and vote for the very best witnesses.

·
·

Cutting to 5-10 votes would also limit collusion/friend possibilities to lock up the top 20-30 positions.

·

I'd like to see a similar concept implemented into reputation scores to actually make reputation an actual viable concept

·
·

I very strongly agree. IRL there is nothing more important than rep, and the gamability of rep on Steemit poorly reflects it's value to society. We see the mechanisms for gaming rep IRL as corruption, and the consequences of gamification of reputation can be existential, both to folks that suffer according it to those that gain it absent merit, and those that are found to abuse it.

Reputation is far more valuable to society than it's current implementation on Steemit recognizes, and potentiating a useful reputation mechanism will increase the utility of Steemit to society.

·

Agreed.

·

I second this

·

Throwing in my "me too" here. This is one of the most important things that can be done for Steem right now, in my opinion.

Thanks for updating us everytime. Scalability is the main issue that the other blockchain are still battling over. There is a saying that Prevention is better than cure. I know in just a matter of time many people including celebrities will get on board and it will exponentially increase the user and the scalability issue will come. Implementing this kind of solution will enhance our user experience in the platform. An update like this is why I do believe that Steem is the best blockchain out there. Great great great update! HF20!

And yet there is still no formula to allow free account registrations without manual approval (SMS verification): The #1 barrier for new users wanting to join STEEM and DTube.

It's a necessary step for proper scaling on the blockchain. Thank you for the weekly update, they have been greatly appreciated.

This has indeed been something that often has been very confusing for new users. While the previous balancing worked (though it sometimes really did get hammered), the idea of a more tangible, easily understood system is a good step to empowering these users to make better choices — based on understanding their own abilities, instead of smashing a button and praying the numbers swing ever in their favour.

This is something I spend a lot of time explaining to users new and old, and to the skeptics out there. I appreciate the move towards something more predictable that will help people better actually understand the technology they enjoy. Cheers.

Last week we started the roll out of steemhost.com. Steemhost shows the blog list of individuals (or groups) but redirects people to steemit or busy to view (and vote) the specific post.

In addition to being able to include the content of specific people on the site, it also adds the element of banner ads advertising. What is important about this is that the advertisements are purchased with steem which should cause upward pressure on its price.
Part of what will motivate steemians to administer their own pages and steem visitors to view those sites are that they are rewarded in steem power delegation.

To view an example of a page where two bloggers are on the same page is
https://steemhost.com/@steemitblog

I set both @steemitblog and @ned to act as contributors to this page.

If you are a visitor, you need to log in with steemconnect to be rewarded.

·

"Error when processing request!" awesome :D

·
·

Could you confirm which browser you are using? Were you attempting the login (it uses steemconnect to only identify who you were. Thanks for the error checking.

·
·
·

i'm using firefox with some security addons i didn't even try to do some thing just clicking the url you provided.

i do use add blockers and cookie blockers and the Noscript add on but still 99% of the website can handle that these days

·
·
·
·

I think I see what the problem is. When you are logged into the public pages they add some additional feature (allowing you to bookmark pages, add additional hosts etc). It also allows you to be rewarded on your click and views. Normally if you are not logged in, it passes all the rewards to the host of the page. While there is no problem with Chrome, the best that I can do is pass the message on to the programmer. We do have a couple of cookies ... mainly associated with remembering who you are. We have some tracking functions ... in order to record what who has visited which pages and when, to prevent people from tapping the F5 key.

There should be a banner ad just above the search button. Possibly your ad blocker doesn't like something if that section is missing.

We have identified that Microsoft Edge doesn't always cooperate but that is on the steemconnect pages ... so it is out of our hands. I have sent this on to tech support so in a day or so, I will post something when it is resolved.

·
·
·
·
·

the add blocker i use is : ublock origin ,
the tracker blocker is : Privacy badger and / or ghostery .
it's firefox 60.02 (64bits) that i used for the link

·
·
·
·
·
·

It might be fixed now. One programmer had applied ssl to the site while another hadn't updated webconfig. Firefox is a little more sensitive. Meanwhile Microsoft Edge is just annoying.

·
·
·
·
·
·
·

i never use edge , it's bad enough that i need windows to play my games .

It sounds to me that administrators and developers will be able to schedule when they undertake certain tasks. For example, if they send out a transaction to all their members, they can do it at a time when RCs are cheap to preserve the application's users to conduct other activitie son the blockchain.

Am I understanding correctly?

·

I think this is pretty much it, just how electricity is cheaper at some places during daytime because people are out for work and its more expensive at night because people have ACs and heaters on. There will be times of the day when it will be cheaper to send batches of transactions when the network is having less use... It will be tricky to figure out since Steem has people from all over the globe but its possible.

It seems to me that this will make for a more familiar experience. Currently, when it comes to times of high blockchain usage, the "You don't have enough bandwidth" is a very unfamiliar problem for most users.

In a lot of freemium apps these days, you can do a particular number of transactions each day for free but you have to buy the premium version if you want anything more than that.

I think that the act of familiarizing the process like this will make for a better user experience because they'll "get it" much more easily.

News and Updates like this should be pinned at the top for at least 24 hours.

Uhm, it sounds technical and I don't fully understand it. So what effect will it have on users again? Is it disadvantageuous to newbies/redfishes and will it affect minnows in a bad way or no?

What about the dust votes issue? Is there anything that will be done about it? Will I be told to submit a proposal on utopian.io again? I'm not into programming and such.

I hope you are right about this...

Summary: These changes will lead to a far more pleasant, more transparent, and more predictable user experience across all Steem applications, and will more accurately allocate network resources by leveraging an efficient market mechanism.

I hope you've focused grouped it, spoke to existing users of all varieties (not just your group of long time friends and large steem holders), and really put some thought into the dreadful user experience that most newbies and minnows feel (and thus have subsequently left). I know it doesn't appear that you are doing these things, but obviously they may be happening and no one talks about it.

Either way, if it results in a smoother experience where "people" are less frustrated and don't have to spend more just to use the system, then I'm all for it... And by "people" I mean real humans that are needed to develop real communities.

I will see and hope you are right about this being a "pleasant" change, I want to see steemit succeed just like the big boys (and girls)!

I think you need to make it easier to make money on steemit, this would increase retention rates of users.

One way to do this would be to Reduce the dust threshold cap from the current 0.0251 to 0.001

most people don't realise you need 200 sp for your vote to actually earn anything.

@steemitblog

Figure out a better approach to deter spammers / bandwidth usage

Transactions should “cost” more during times when the blockchain is busy and less during times when it is not.

Shouldn't an architecture work the other way? when the system is optimally used then you have budgeted for the transactions already and hence it must be cheaper. When it is not often used, then it becomes more expensive because of dark resources lying around. as far as the user is concerned, more the number of transactions, the more he is subsidized to use it.

When a user can know how many RCs they will have, how long it takes for those RCs to recharge, and how many RCs each transaction will cost, they will be able to make informed decisions about how to spend their RCs.

Earlier we were faced with unknown bandwidth failures. Now we will be explicitly told whether we can post or not. I guess makes it easy for me to not be baffled by strange failures. This change is about educating the user about the problem and not solving anything. unless there is something thought of in the future that i am not aware of.

The most important thing to understand about RCs is that they will be non-transferable credits given to each account based on how much Steem Power it has,

we are already aware of steem power. why not just stick with it instead? internally RCs are a good idea but is there any significant problem that we are solving by pushing the information to the average user?

RCs will be the “currency” that the blockchain uses to price the resources in those stockpiles, and the internal medium of exchange for said resources.

did this change come about because of our progression to acquiring new communities using SMTs? i think that this a great idea if you can expose the blockchain to enterprise customers. one of the largest problems that enterprises face is the dissatisfaction of their customers with literature (training manuals, installation docs...). get enterprises to use our blockchain for content publishing and get their customers to constantly rate and correct them. then RCs are a beautiful approach.

i think the RC system has far better applications if they are outward facing rather than within the existing community. i understand the rationale though. good luck

Somtimes i get lost in the technical details but thanks for making this one pretty easily understandable.
Just another system to replace bandwidth ... I never understood how bandwidth works but sounds like this one will be understandable (even if complex) ... I'm guessing this will make HF20 a bit more important to implement sooner than later. So that's always fun.

MY QUESTION:
Will RCs be spent just by users or will tools doing API calls (or other backend stuff looking at the blockchain) need a stockpile of RCs?

·

Any account (whether bot or human) that posts a transaction to the blockchain will be charged RCs.

·
·

OK no thanks... so checking out the blockchain or doing a call to search it is no cost... just those that POST. Just clarifying.

If the changes in the updated privacy and TOS pages were made due to this update, you need to look at that update and how it was implemented, 5 seconds is insufficient time to read a multi page document, I was forced to accept changes to these two documents in order to continue using steemit. So if the new updated pages were a result of this please rework the Update and allow others time to read before having to blindly accept policy changes.

I know this had absolutely nothing to do with the changes in policy, but no other avenue appears to be available to let you know it was a screwed up update.

·

It looks like the TOS change was rolled back pretty quickly after it was applied.

Great job, guys! Thanks for the work, let's do it ;)

Markets, after all, are arguably humanity’s most practical method of distributing resources among populations beyond the bounds of a small tribe.

Just a little bit of food for throught thrown in there for the reader's amusement? ;p

Exciting update. I hope this does indeed make spamming more expensive / demanding of SP.

The gamification angle is fascinating to me and will be very excited to see how that idea matures.

Love the idea of rewarding people for growing their account in more ways than just blogging / creating and curating content. This seems to tackle investing in the blockchain and bring that element into growing their account.

If you can effectively gamify it...Very exciting :)

buen post amigo

Mantap...

good job

The first thing that comes to mind is "How is freedom and decentralization?", Although on the other hand this will solve the problem with spam.

Первое что приходит на ум - "Как же свобода и децентрализация?", хотя, с другой стороны так будет решена проблема со спамом.

Enabling the formation of a mental model is critical to satisfying our third design requirement: more predictable user experiences on Steem.

Mental Model.. Mental Model.. Mental Map.... Ah! ¡Stigma!

Uhm, ¿Where the heck I've heard that before?

While a primary objective for Steem is to minimize the cognitive load on users, this system will create new opportunities for developers looking to create more engaging applications.

Cognitive load.. Cognitive load.. Engaging applications....

Uhm, erm yeah, better leave that like that!! };)

RC~SP & “stockpile”~"rewards pool" ¿Is that right? Yeah, discuss!!

How are the RC costs calculated, sounds to me like a fixed fee? I like the idea of beeing hardware near at the cost, but if RAM prices for example keep rising, these costs need to be increased. Or did I get something wrong here?

Sounds good if it helps deal with spam and other abuse. Are all the microtransactions by people doing services an issue? There's also lots of tiny votes by voting trails. I don't see the a 1% vote by a minnow will gain the anything anyway. If a user cannot perform an action then they need good feedback. Some of the error messages I see are still a bit cryptic. I hope this work does not push hf20 back as we need other features it brings, but I appreciate you want to minimise the number of big releases

Why is payout declined. I never seen that one before

·

as a publisher you have that choice. the stories from @steemitblog and @ned often times decline payout.

·

"Declined payout" means that the author doesn't want the payout generated from the upvotes. His or her payout will be returned to the rewards pool (or rather not distributed at all).

On the other hand, a high payout, even if it's declined, can put the post on the trending page.

Look interesting and probably is an improvement. Otherwise, you wouldn't be pondering about it and developing it.

For me, the bandwidth model is (was) a non-issue. I never even tried to understand it and it never was a problem. It might be a different story for new users with a low amount of SP, though.

Will this GC approach in any way force the change of behavior of an average user?

Ps: I love the usage of high-speech in some sentences. Looks like you developers are also philosophers and psychologists at the same time :)

Maybe you had a different idea in mind, about the plan after AFTER HF20, regarding bandwidth.

Here's how I see things unfold, but I may be wrong.

RCs will continue to be shown as a progress bar on the tools which currently show the bandwidth.

All apps and tools will need to update to interactively add information (as tooltips?) about the cost for each action.

Am I right? I realize these are two separate things, blockchain development and apps/tools development, but how else would these new costs be presented without creating confusion?

I have not understood everything, but at least the developer team is working hard to optimize the bandwidth and increase the user experience.

Should be great to see a video of this feature in action!

First of all, do you realize that if you explain HF20 to 1,000 randomly chosen people in a crowded shopping street, you will have 99% chance that all of these 1,000 people don't understand 1 word of what you have written. Does anyone realize that?

And how many people on Steemit do actually understand this?

And second, what happened to the curation adjustment? I thought HF20 would contain a change for how curation works?

·

That was explained on a previous post, this is just an update to the bandwidth usage and i don't think your argument of explaining it to 1000 people is valid, try to explain to people how electricity works to 1000 random people and no one will understand you, this is advanced stuff, the curation stuff you are looking for was posted awhile ago.

·
·

To put my story a bit better in perspective, I think many people believed that the banking crises became reality because those bankers were selling very complex products. And most bankers didn't even understand how those complex products worked. So it looks like the same thing is happening again, but this time with crypto.

Besides that I see many other things happening in crypto, that we also saw with bankers. The most clear ones are greed and scams.

About electricity, you have a fair point here. Nevertheless it is not an investment. It works as simple as: what you use is what you pay for. You can check every day what you have used.

If I would delegate to Steemhunt, I would be really clueless about what I'm doing. That goes for the majority of people. I see many crypto enthousiastics that say this will become mainstream one day. But if anyone learned something from the banking crises, it better won't become mainstream.

I hope this clarifies things a bit?

·

There are numerous other changes in HF20. This post is only discussing bandwidth.

·
·

Do you happen to know when HF19 took place? And when HF20 was first time announced?

·
·
·

I think HF20 was announced about 5 years ago, but I could be off by a few months ;)

·
·
·
·

That's usually what happens when the dead keep talking 😜

·
·
·
·
·

You hear them too?

Well, in any case, Elvis Presley says "hi".

·
·
·

HF20 was proposed June 22 of last year: https://steemit.com/steemit/@steemitblog/proposing-hardfork-0-20-0-velocity

HF19 activated around that time I believe, but that is from memory so I could be off by a bit.

·
·
·
·

Thanks!

Your brains seem to work very smooth!

This long ago announced HF20 confirms for me that this go way to slowly. This is not the flexibility this platform needs to have for the long term.

To bad I didn't read anything about curation rewards update. I really thought I remember that multiple Steemonians wrote the 30 minute rule will become a 15 minute rule.

·
·
·
·
·

I believe the (still) planned change to curation was added after the original announcement.

I would expect when there is a release candidate, there will be a new post describing all the changes.

Congratulations @steemitblog!
Your post was mentioned in the Steemit Hit Parade in the following category:

  • Pending payout - Ranked 4 with $ 493,7

hopefully this new formula of bandwith will make steemit better and so the users more comfortable in using steemit. thank you so much for the steemit blockchain team. i proud to be here

To listen to the audio version of this article click on the play image.

Brought to you by @tts. If you find it useful please consider upvoting this reply.

although I like the idea, I do see some drawbacks. Mainly the use of what you have termed another 'currency'. We have steem, we have sbd, we have vest, and now we will have RC.

This opens a question...will smts need SP or RC to run smoothly? SP will give you RC, so its still SP that is needed right?

·

I was thinking the same thing.

If it's proportional to SP, why not just use a portion of the voting power rewards since that's already built in?

You mention that Rc's will recharge over time. Can you elaborate exactly how?

Congrats, you made the #steemitminute for today!

Click the Image Below to see the Video!

I am more concerned about "Vote Buying Bots"

Cant you make Vote Buying Illegal so the "Rich dont get richer" here

Please don't display yet another currency for the poor users! Could you visualize the RCs as a progress bar instead?

·

It will be entirely up to the UIs to decide how to best represent the data to the user.

Good to know. Sorry I missed this for so long.
Could we look at enabling editing of posts after payout, perhaps with a high RC cost?
We'd only use it rarely, but looking back through my blog and seeing broken links really bothers me. I can't be the only one.