https://github.com/gridcoin-community/Gridcoin-Research/releases/tag/220.127.116.11 submitted by
Finally! After over ten months of development and testing, "Fern" has arrived! This is a whopper. 240 pull requests merged. Essentially a complete rewrite that was started with the scraper (the "neural net" rewrite) in "Denise" has now been completed. Practically the ENTIRE Gridcoin specific codebase resting on top of the vanilla Bitcoin/Peercoin/Blackcoin vanilla PoS code has been rewritten. This removes the team requirement at last (see below), although there are many other important improvements besides that.
Fern was a monumental undertaking. We had to encode all of the old rules active for the v10 block protocol in new code and ensure that the new code was 100% compatible. This had to be done in such a way as to clear out all of the old spaghetti and ring-fence it with tightly controlled class implementations. We then wrote an entirely new, simplified ruleset for research rewards and reengineered contracts (which includes beacon management, polls, and voting) using properly classed code. The fundamentals of Gridcoin with this release are now on a very sound and maintainable footing, and the developers believe the codebase as updated here will serve as the fundamental basis for Gridcoin's future roadmap.
We have been testing this for MONTHS on testnet in various stages. The v10 (legacy) compatibility code has been running on testnet continuously as it was developed to ensure compatibility with existing nodes. During the last few months, we have done two private testnet forks and then the full public testnet testing for v11 code (the new protocol which is what Fern implements). The developers have also been running non-staking "sentinel" nodes on mainnet with this code to verify that the consensus rules are problem-free for the legacy compatibility code on the broader mainnet. We believe this amount of testing is going to result in a smooth rollout.
Given the amount of changes in Fern, I am presenting TWO changelogs below. One is high level, which summarizes the most significant changes in the protocol. The second changelog is the detailed one in the usual format, and gives you an inkling of the size of this release.
Note that the protocol changes will not become active until we cross the hard-fork transition height to v11, which has been set at 2053000. Given current average block spacing, this should happen around October 4, about one month from now.
Note that to get all of the beacons in the network on the new protocol, we are requiring ALL beacons to be validated. A two week (14 day) grace period is provided by the code, starting at the time of the transition height, for people currently holding a beacon to validate the beacon and prevent it from expiring. That means that EVERY CRUNCHER must advertise and validate their beacon AFTER
the v11 transition (around Oct 4th) and BEFORE
October 18th (or more precisely, 14 days from the actual date of the v11 transition). If you do not advertise and validate your beacon by this time, your beacon will expire and you will stop earning research rewards until you advertise and validate a new beacon. This process has been made much easier by a brand new beacon "wizard" that helps manage beacon advertisements and renewals. Once a beacon has been validated and is a v11 protocol beacon, the normal 180 day expiration rules apply. Note, however, that the 180 day expiration on research rewards has been removed with the Fern update. This means that while your beacon might expire after 180 days, your earned research rewards will be retained and can be claimed by advertising a beacon with the same CPID and going through the validation process again. In other words, you do not lose any earned research rewards if you do not stake a block within 180 days and keep your beacon up-to-date.
The transition height is also when the team requirement will be relaxed for the network.
Besides the beacon wizard, there are a number of improvements to the GUI, including new UI transaction types (and icons) for staking the superblock, sidestake sends, beacon advertisement, voting, poll creation, and transactions with a message. The main screen has been revamped with a better summary section, and better status icons. Several changes under the hood have improved GUI performance. And finally, the diagnostics have been revamped.
The wallet sync speed has been DRASTICALLY improved. A decent machine with a good network connection should be able to sync the entire mainnet blockchain in less than 4 hours. A fast machine with a really fast network connection and a good SSD can do it in about 2.5 hours. One of our goals was to reduce or eliminate the reliance on snapshots for mainnet, and I think we have accomplished that goal with the new sync speed. We have also streamlined the in-memory structures for the blockchain which shaves some memory use.
There are so many goodies here it is hard to summarize them all.
I would like to thank all of the contributors to this release, but especially thank @cyrossignol, whose incredible contributions formed the backbone of this release. I would also like to pay special thanks to @barton2526, @caraka, and @Quezacoatl1, who tirelessly helped during the testing and polishing phase on testnet with testing and repeated builds for all architectures.
The developers are proud to present this release to the community and we believe this represents the starting point for a true renaissance for Gridcoin!
Most significantly, nodes calculate research rewards directly from the magnitudes in EACH superblock between stakes instead of using a two- or three- point average based on a CPID's current magnitude and the magnitude for the CPID when it last staked. For those long-timers in the community, this has been referred to as "Superblock Windows," and was first done in proof-of-concept form by @denravonska.
- Network magnitude unit pinned to a static value of 0.25
- Max research reward allowed per block raised to 16384 GRC (from 12750 GRC)
- New CPIDs begin accruing research rewards from the first superblock that contains the CPID instead of from the time of the beacon advertisement
- 500 GRC research reward limit for a CPID's first stake
- 6-month expiration for unclaimed rewards
- 10-block spacing requirement between research reward claims
- Rolling 5-day payment-per-day limit
- Legacy tolerances for floating-point error and time drift
- The need to include a valid copy of a CPID's magnitude in a claim
- 10-block emission adjustment interval for the magnitude unit
- One-time beacon activation requires that participants temporarily change their usernames to a verification code at one whitelisted BOINC project
- Verification codes of pending beacons expire after 3 days
- Self-service beacon removal
- Burn fee for beacon advertisement increased from 0.00001 GRC to 0.5 GRC
- Rain addresses derived from beacon keys instead of a default wallet address
- Beacon expiration determined as of the current block instead of the previous block
- The ability for developers to remove beacons
- The ability to sign research reward claims with non-current but unexpired beacons
As a reminder:
- Beacons expire after 6 months pass (180 days)
- Beacons can be renewed after 5 months pass (150 days)
- Renewed beacons must be signed with the same key as the original beacon
- Magnitudes less than 1 include two fractional places
- Magnitudes greater than or equal to 1 but less than 10 include one fractional place
- A valid superblock must match a scraper convergence
- Superblock popularity election mechanics
- Yes/no/abstain and single-choice response types (no user-facing support yet)
- To create a poll, a maximum of 250 UTXOs for a single address must add up to 100000 GRC. These are selected from the largest downwards.
- Burn fee for creating polls scaled by the number of UTXOs claimed
- 50 GRC for a poll contract
- 0.001 GRC per claimed UTXO
- Burn fee for casting votes scaled by the number of UTXOs claimed
- 0.01 GRC for a vote contract
- 0.01 GRC to claim magnitude
- 0.01 GRC per claimed address
- 0.001 GRC per claimed UTXO
- Maximum length of a poll title: 80 characters
- Maximum length of a poll question: 100 characters
- Maximum length of a poll discussion website URL: 100 characters
- Maximum number of poll choices: 20
- Maximum length of a poll choice label: 100 characters
- Magnitude, CPID count, and participant count poll weight types
- The ability for developers to remove polls and votes
[18.104.22.168] 2020-09-03, mandatory, "Fern"
- Backport newer uint256 types from Bitcoin #1570 (@cyrossignol)
- Implement project level rain for rainbymagnitude #1580 (@jamescowens)
- Upgrade utilities (Update checker and snapshot downloadeapplication) #1576 (@iFoggz)
- Provide fees collected in the block by the miner #1601 (@iFoggz)
- Add support for generating legacy superblocks from scraper stats #1603 (@cyrossignol)
- Port of the Bitcoin Logger to Gridcoin #1600 (@jamescowens)
- Implement zapwallettxes #1605 (@jamescowens)
- Implements a global event filter to suppress help question mark #1609 (@jamescowens)
- Add next target difficulty to RPC output #1615 (@cyrossignol)
- Add caching for block hashes to CBlock #1624 (@cyrossignol)
- Make toolbars and tray icon red for testnet #1637 (@jamescowens)
- Add an rpc call convergencereport #1643 (@jamescowens)
- Implement newline filter on config file read in #1645 (@jamescowens)
- Implement beacon status icon/button #1646 (@jamescowens)
- Add gridcointestnet.png #1649 (@caraka)
- Add precision to support magnitudes less than 1 #1651 (@cyrossignol)
- Replace research accrual calculations with superblock snapshots #1657 (@cyrossignol)
- Publish example gridcoinresearch.conf as a md document to the doc directory #1662 (@jamescowens)
- Add options checkbox to disable transaction notifications #1666 (@jamescowens)
- Add support for self-service beacon deletion #1695 (@cyrossignol)
- Add support for type-specific contract fee amounts #1698 (@cyrossignol)
- Add verifiedbeaconreport and pendingbeaconreport #1696 (@jamescowens)
- Add preliminary testing option for block v11 height on testnet #1706 (@cyrossignol)
- Add verified beacons manifest part to superblock validator #1711 (@cyrossignol)
- Implement beacon, vote, and superblock display categories/icons in UI transaction model #1717 (@jamescowens)
- neuralnet: Add integrity checking to researcher accrual snapshot registry #1727 (@jamescowens)
- Add workaround for scrypt assembly on macOS #1740 (@cyrossignol)
- gui: Build onboarding/beacon wizard #1739 (@cyrossignol)
- doc: Add CONTRIBUTING.md from bitcoin #1723 (@div72)
- rpc: Implement inspectaccrualsnapshot and parseaccrualsnapshotfile #1744 (@jamescowens)
- scraper: Add disk based state backing for verified beacon list in scraper #1751 (@jamescowens)
- Add ability to recover beacon in block version 11+ #1768 (@cyrossignol)
- refactor: Add transaction context to contract handlers #1777 (@cyrossignol)
- gui: Add context for when BOINC is attached to a pool #1775 (@cyrossignol)
- doc: Clarify what to do if PR in multiple categories (for CONTRIBUTING.md) #1798 (@RoboticMind)
- qt: Add option to choose not to start the wallet minimized #1804 (@jamescowens)
- superblock: Add check for OutOfSyncByAge to SuperblockValidator::Validate #1806 (@jamescowens)
- contract: Standardize contract validation and add block context #1808 (@cyrossignol)
- add seed.gridcoin.pl to default config #1812 (@wilkart)
- gui: Implement sidestake send display #1813 (@jamescowens)
- gui: Add pool/investor pages to researcher wizard #1819 (@cyrossignol)
- ci: Port lint scripts from Bitcoin #1823 (@div72)
- doc: Create basic readme in contrib #1826 (@RoboticMind)
- gui: Implement TransactionRecord::Message #1829 (@jamescowens)
- rpc: Add private_key_available to beaconstatus #1833 (@a123b)
- gui: Validate email address in researcher wizard #1840 (@a123b)
- rpc: Add "getrawwallettransaction" RPC function #1842 (@cyrossignol)
- consensus: Set block version 11 threshold height for mainnet #1862 (@cyrossignol)
- Upgrade LevelDB from v1.17 to v1.20 #1562 (@cyrossignol)
- Re-enable scrypt optimizations #1450 (@denravonska)
- Derive CScript from prevector type (optimization) #1554 (@cyrossignol)
- Disable quorum for grandfathered blocks to speed up sync #1568 (@cyrossignol)
- Refactor hashBoinc for binary claim contexts #1558 (@cyrossignol)
- integrated_scraper_2 branch tracking PR #1559 (@jamescowens)
- Upgrade depends - OpenSSL to 1.1.1d #1581 (@jamescowens)
- Ubuntu 19.10 fixes #1590 (@denravonska)
- Force a re-parse of legacy claims in generated blocks #1592 (@cyrossignol)
- Improve the "versionreport" RPC output #1595 (@cyrossignol)
- Overhaul the core tally and accrual system #1583 (@cyrossignol)
- Overhaul the superblock quorum system #1597 (@cyrossignol)
- Add more data to the "superblocks" RPC output #1599 (@cyrossignol)
- Update Windows Build doc #1606 (@barton2526)
- Change the order of calls in gridcoinresearchd.cpp to optimize rpc shunt path #1610 (@jamescowens)
- Change staking tooltip to display frequency #1611 (@jamescowens)
- Enhancements to ETTS #1442 (@jamescowens)
- Standardize money values as integers #1614 (@cyrossignol)
- Clean up and optimize legacy coin age code #1616 (@cyrossignol)
- Some scraper cleanups #1620 (@jamescowens)
- Reorganize accrual code and fix 6-month cutoff #1630 (@cyrossignol)
- Update Copyright years #1633 (@barton2526)
- Change team whitelist delimiter to <> for CPID detection #1634 (@cyrossignol)
- Change team whitelist separator to <> to accomodate more team names #1632 (@jamescowens)
- Change Curl download speed type to support older environments #1640 (@cyrossignol)
- Optimize logo SVGs used for tray icons #1638 (@cyrossignol)
- Tweak consolidateunspent rpc function #1644 (@jamescowens)
- ETTS and staking icon enhancements #1650 (@jamescowens)
- Implement new transaction fees for block version 11 #1652 (@jamescowens)
- Optimize in-memory storage of superblock data #1653 (@cyrossignol)
- Miscellaneous superblock API improvements and housekeeping #1654 (@cyrossignol)
- Update openssl to 1.1.1f compatibility #1660 (@jamescowens)
- Optimize bdb to avoid synchronous flush of database #1659 (@jamescowens)
- Add support for CPID input to "lifetime" RPC function #1668 (@cyrossignol)
- Overhaul the contract handling system #1669 (@cyrossignol)
- Make the autostart mainnet/testnet aware #1671 (@jamescowens)
- Remove slashes from User Agent in peers tab #1674 (@div72)
- Refactor contracts for polymorphic binary payloads #1676 (@cyrossignol)
- Overhaul the beacon system #1678 (@cyrossignol)
- Replace boost::optional with non-owning pointers #1680 (@cyrossignol)
- Optimize proof-of-stake validation #1681 (@cyrossignol)
- Updated Slack link #1683 (@NeuralMiner)
- Update build-unix.md #1686 (@Quezacoatl1)
- Replace deprecated QT methods #1693 (@Pythonix)
- Made protocol.h more similar to bitcoin #1688 (@Pythonix)
- Touch up some details for block version 11 #1697 (@cyrossignol)
- More tweaks for block version 11 #1700 (@cyrossignol)
- Finish the conversion to the BCLog class based logger #1699 (@jamescowens)
- Move claim version transitional code in miner for proper signature #1712 (@cyrossignol)
- doc: Update threads in coding.txt #1730 (@div72)
- qt: Include QPainterPath in trafficgraphwidget.cpp #1733 (@div72)
- doc: Update doc/build-unix.md #1731 (@div72)
- gui: Show peers tab on connections icon click #1734 (@div72)
- refactor: Change return type of IsMine to isminetype && move wallet files to wallet directory #1722 (@div72)
- build: Updates boost to 1.73.0 for depends #1673 (@jamescowens)
- doc: Update Unit Test Readme #1743 (@RoboticMind)
- wallet: Change Assert To Error Message In kernel.cpp #1748 (@RoboticMind)
- scraper: Shorten display representation of verification codes #1754 (@cyrossignol)
- log: Change ".B." to Clear Message #1758 (@RoboticMind)
- util: Fix braindamage in GetDefaultDataDir() #1737 (@jamescowens)
- scraper: Improve scraper processing of beacon verifications #1760 (@jamescowens)
- scraper: Add instrumentation to convergencereport #1763 (@jamescowens)
- rpc: Improve rpc stress test script #1767 (@tunisiano187)
- Generalize enum serialization #1770 (@cyrossignol)
- scraper: Improve handling of ETags in http class and tweak verified beacon logic #1776 (@jamescowens)
- scraper: Improve ProcessNetworkWideFromProjectStats and other tweaks #1778 (@jamescowens)
- researcher: Automate beacon advertisement for renewals only #1781 (@cyrossignol)
- gui: Tweak behavior of beacon page in researcher wizard #1784 (@cyrossignol)
- Prepare for block version 11 hard-fork on testnet #1787 (@cyrossignol)
- scraper: Modify UpdateVerifiedBeaconsFromConsensus #1791 (@jamescowens)
- gui: Optimize OverviewPage::updateTransactions() #1794 (@jamescowens)
- ci: Adopt ci changes from Bitcoin #1795 (@div72)
- consensus: switch snapshot accrual calculation to integer arithmetic #1799 (@cyrossignol)
- voting: Overhaul the voting system #1809 (@cyrossignol)
- contract: Optimize contract replay after chain reorganization #1815 (@cyrossignol)
- contract: Reimplement transaction messages as contracts #1816 (@cyrossignol)
- staking: Sign claim contracts with coinstake transaction #1817 (@cyrossignol)
- gui: Change research wizard text #1820 (@div72)
- net: Update protocol version and clean up net messaging #1824 (@cyrossignol)
- rpc, wallet: Corrections to GetAmounts #1825 (@jamescowens)
- gui: Tweak some minor researcher wizard details #1830 (@cyrossignol)
- gui: Change GetEstimatedStakingFrequency text #1836 (@jamescowens)
- scraper: Scraper global statistics cache optimization #1837 (@jamescowens)
- doc: Update Vulnerability Response Process #1843 (@RoboticMind)
- scraper: Optimization of manifest and parts sharing between ConvergedScraperStatsCache, mapManifest, and mapParts #1851 (@jamescowens)
- consensus: Update Checkpoints #1855 (@barton2526)
- docs: Update docs to build off master #1856 (@barton2526)
- gui: Fix and improve GUI combo box styles #1858 (@cyrossignol)
- build: Tweak Gridcoin installer for Fern release #1863 (@jamescowens)
- Remove old research age checks (rebase #1365) #1572 (@cyrossignol)
- Remove PrimaryCPID check from diagnostics dialog #1586 (@cyrossignol)
- Remove missed label for PrimaryCPID from diagnostics #1588 (@cyrossignol)
- Remove legacy quorum messaging system (@neural network) #1589 (@cyrossignol)
- Remove old remnants of legacy smart contract experiments #1594 (@cyrossignol)
- Remove block nonce for version 11 #1622 (@cyrossignol)
- Delete obsolete contrib/Installer and Upgrader directories #1623 (@jamescowens)
- Remove redundant LoadAdminMessages() calls #1625 (@cyrossignol)
- Remove some legacy informational RPC commands #1658 (@cyrossignol)
- Remove informational magnitude field from binary claims #1661 (@cyrossignol)
- Remove fDebug3,4, and net and convert to BCLog::LogFlags #1663 (@jamescowens)
- Remove qt5.7.1 depends support build System #1665 (@iFoggz)
- Remove unused jQuery library #1679 (@cyrossignol)
- Remove unused NetworkTimer() function and global state #1701 (@cyrossignol)
- Refactor claim context objects into contracts #1704 (@cyrossignol)
- Clean old assets up #1718 (@div72)
- Remove legacy "rain" RPC (not by-project rain) #1742 (@cyrossignol)
- Temporarily disable voting system on testnet #1769 (@cyrossignol)
- gui: Remove legacy GUI transaction description for contracts #1772 (@cyrossignol)
- gui: Remove transaction fee setting #1780 (@cyrossignol)
- trivial: Cleanup unused legacy functions #1793 (@cyrossignol)
- mining, rpc: Remove kernel-diff-best and kernel-diff-sum #1796 (@jamescowens)
- refactor: Remove libs subdirectory #1802 (@div72)
- scraper: cleanup unused/unnecessary functions #1803 (@jamescowens)
- gui: Remove useless "Detach databases at shutdown" #1810 (@jamescowens)
- test: Remove testnet condition for standard transactions #1814 (@cyrossignol)
- consensus: Remove transitional testnet code #1854 (@cyrossignol)
- Fix "Owed" amount in output of "magnitude" RPC method #1569 (@cyrossignol)
- Add support for paths with special characters on Windows #1571 (@cyrossignol)
- Fix lingering peers.dat temp files and clean up remaining paths #1582 (@cyrossignol)
- Fix incorrect beacon length warning in GUI transaction list #1585 (@cyrossignol)
- Fix default config file line endings on Windows #1587 (@cyrossignol)
- Reenable Travis builds for MacOS #1591 (@jamescowens)
- Correct peer detail info background color #1593 (@jamescowens)
- Fix exception in debug3 mode #1598 (@cyrossignol)
- Fix deadlock in "getmininginfo" RPC function #1596 (@cyrossignol)
- Fix accuracy of statistics in "network" RPC output #1602 (@cyrossignol)
- Fix heights for quorum vote weight calculations #1604 (@cyrossignol)
- Fix deadlock in log archiver when rename fails #1607 (@cyrossignol)
- Fix a spurious segmentation fault during client load on Windows with fast CPUs #1608 (@jamescowens)
- Fix lock order debugging and potential deadlocks #1612 (@jamescowens)
- Add dependencies #1613 (@Scalextrix)
- Fix std namespace pollution #1617 (@denravonska)
- Add missing condition for newbie accrual computer #1618 (@cyrossignol)
- Track first reward blocks in research accounts #1619 (@cyrossignol)
- Fix lingering beacon warning after advertisement #1627 (@cyrossignol)
- Fix accrual calculation for new, zero-magnitude CPIDs #1636 (@cyrossignol)
- Fix diagnostics, add ETTS test, fix tooltipcolor, add missing lock, and add email=investor check #1647 (@jamescowens)
- Fix help message of two RPC methods #1656 (@div72)
- Fix legacy accrual for newbie with non-zero past reward #1667 (@cyrossignol)
- Fix GUI autostart on Windows for paths with wide characters #1670 (@cyrossignol)
- Qualify boost bind placeholders with their full namespace #1672 (@Ponce)
- Fix suffix when copying txids #1677 (@div72)
- Unnecessary if-statement removed #1685 (@Pythonix)
- Fix consolidatemsunspent Help Message #1687 (@Pythonix)
- Fix gettransaction help message #1691 (@Pythonix)
- Fix GetNewMint To Look for Stakes #1692 (@RoboticMind)
- Suppress deprecated copy warnings for Qt with GCC 9+ #1702 (@cyrossignol)
- Fix exclusion error on stats processing and misplaced ENDLOCK logging entry #1710 (@jamescowens)
- Removed unnecessary comparison #1708 (@Pythonix)
- Fixed typo #1707 (@Pythonix)
- Fix out-of-bounds exception for peers tab version slashes #1713 (@cyrossignol)
- Fix transition for v1 superblocks when reorganizing #1714 (@cyrossignol)
- Touch up transition to version 2 transactions #1715 (@cyrossignol)
- Avoid mutating transactions in ConnectBlock() #1716 (@cyrossignol)
- Skip beacon advertisement when already pending #1726 (@cyrossignol)
- Fix Windows cross-compilation in newer environments #1728 (@cyrossignol)
- Fix out-of-bounds access in IsMineInner() #1736 (@cyrossignol)
- Fix a couple of block version 11 issues #1738 (@cyrossignol)
- Fix null pointer dereference in GUI researcher model #1741 (@cyrossignol)
- accrual: Reset research accounts when rebuilding accrual snapshots #1745 (@cyrossignol)
- scraper: Correct update for verified beacons #1747 (@jamescowens)
- accrual: Refactor tally initialization for snapshot rebuild #1749 (@cyrossignol)
- rpc: Fix "cpid" field in "beaconconvergence" RPC output #1750 (@cyrossignol)
- accrual: Fix snapshot accrual superblock state transitions #1752 (@cyrossignol)
- scraper: Correct stale verified beacon logic #1753 (@jamescowens)
- rpc: Correct possible divide by zero in getblockstats #1755 (@jamescowens)
- gui: Fix issues with researcher wizard flow #1756 (@cyrossignol)
- wallet: Stop Error When Starting From Zero #1759 (@RoboticMind)
- Don't count empty email as explicit investor #1761 (@cyrossignol)
- accrual: Fix snapshot accrual superblock state transitions #1764 (@cyrossignol)
- rpc: Cleanup Help Message and Fix Typo #1771 (@RoboticMind)
- scraper: Fix scraper etag header case sensitivity #1773 (@cyrossignol)
- consensus: Use explicit time to check if superblock needed #1774 (@cyrossignol)
- gui: Fix scroll area dark theme styles #1785 (@cyrossignol)
- rpc, gui: Fix three divide by zero possibilities #1789 (@jamescowens)
- rpc: Fix balance pre-check in "rainbymagnitude" RPC #1792 (@cyrossignol)
- accrual: Fix outdated comment and correct grammar #1800 (@RoboticMind)
- gui: Fix stuck cursor on labels #1801 (@div72)
- beacon: Fix research wizard beacon renewal status #1805 (@cyrossignol)
- gui: Fix translations for port numbers #1818 (@cyrossignol)
- util: Create parent directory #1821 (@div72)
- mining: Fix coinstake/claim signature order #1828 (@cyrossignol)
- voting: Remove double increment in loop #1831 (@cyrossignol)
- neuralnet, scraper: Fix compilation with gcc5 and older libcurl #1832 (@a123b)
- wallet: Fix smallest coin selection for contracts #1841 (@cyrossignol)
- gui: Fix display of polls with no votes yet #1844 (@cyrossignol)
- gui: add indentation to diagnostic status bar labels #1849 (@jamescowens)
- voting, gui: Fix formatting and alignment of vote shares and percent #1850 (@jamescowens)
- wallet, rpc: Fix for self-transactions in listtransactions #1852 (@jamescowens)
- accrual: Clear any accrual snapshots when syncing from pre-v11 #1853 (@cyrossignol)
- accrual: Fix reset of accrual directory if starting sync below research age height #1857 (@jamescowens)
- gui: Fix researcher wizard layout on macOS with native theme #1860 (@cyrossignol)
Taproot! Everybody wants to have it, somebody wants to make it, nobody knows how to get it!
(If you are asking why everybody wants it, see: Technical: Taproot: Why Activate?
(Pedants: I mostly elide over lockin times)
Briefly, Taproot is that neat new thing that gets us:
- Multisignatures (n-of-n, k-of-n) that are just 1 signature (1-of-1) in length!! (MuSig/Schnorr)
- Better privacy!! If all contract participants can agree, just use a multisignature. If there is a dispute, show the contract publicly and have the Bitcoin network resolve it (Taproot/MAST).
- Activation lets devs work get back to work on the even newer stuff like!!!
- Cross-input signature aggregation!! (transaction with multiple inputs can have a single signature for all inputs) --- needs Schnorr, but some more work needed to ensure that the interactions with SCRIPT are okay.
- Block validation - Schnorr signatures for all taproot spends in a block can be validated in a single operation instead of for each transaction!! Speed up validation and maybe we can actually afford to increase block sizes (maybe)!!
- SIGHASH_ANYPREVOUT - you know, for Decker-Russell-Osuntokun ("eltoo") magic!!!
- OP_CHECKTEMPLATEVERIFY - vaulty vaults without requiring storing signatures, just transaction details!!
So yes, let's activate taproot!
The SegWit Wars
The biggest problem with activating Taproot is PTSD from the previous softfork, SegWit. Pieter Wuille, one of the authors of the current Taproot proposal, has consistently held the position that he will not discuss activation, and will accept whatever activation process is imposed on Taproot. Other developers have expressed similar opinions.
So what happened with SegWit activation that was so traumatic? SegWit used the BIP9 activation method. Let's dive into BIP9!
Basically, BIP9 has a bunch of parameters:
- bit - A field in the block header, the nVersion, has a number of bits. By setting a particular bit, the miner making the block indicates that it has upgraded its software to support a particular soft fork. The bit parameter for a BIP9 activation is which bit in this nVersion is used to indicate that the miner has upgraded software for a particular soft fork.
- timeout - a time limit, expressed as an end date. If this timeout is reached without sufficient number of miners signaling that they upgraded, then the activation fails and Bitcoin Core goes back to the drawing board.
Now there are other parameters (name
) but they are not anywhere near as important as the above two.
A number that is not
a parameter, is 95%. Basically, activation of a BIP9 softfork is considered as actually succeeding if at least 95% of blocks in the last 2 weeks had the specified bit
in the nVersion set. If less than 95% had this bit set before the timeout
, then the upgrade fails and never goes into the network. This is not a parameter: it is a constant defined by BIP9, and developers using BIP9 activation cannot change this.
So, first some simple questions and their answers:
- Why not just set a day when everyone starts imposing the new rules of the softfork?
- This was done classically (in the days when Satoshi was still among us). But this might argued to put too much power to developers, since there would be no way to reject an upgrade without possible bad consequences. For example, developers might package an upgrade that the users do not want, together with vital security bugfixes. Either you live without vital security bugfixes and hire some other developers to fix it for you (which can be difficult, presumably the best developers are already the ones working on the codebase) or you get the vital security bugfixes and implicitly support the upgrade you might not want.
- Sure, you could fork the code yourself (the ultimate threat in the FOSS world) and hire another set of developers who aren't assholes to do the dreary maintenance work of fixing security bugs, but Bitcoin needs strong bug-for-bug compatibility so everyone should really congregate around a single codebase.
- Basically: even the devs do not want this power, because they fear being coerced into putting "upgrades" that are detrimental to users. Satoshi got a pass because nobody knew who he was and how to coerce him.
- Why 95%?
- Suppose the threshold were lower, like 51%. If so, after activation, somebody can disrupt the Bitcoin network by creating a transaction that is valid under the pre-softfork rules, but are invalid under the post-softfork rules. Upgraded nodes would reject it, but 49% of miners would accept it and include it in a block (which makes the block invalid) And then the same 49% would accept the invalid block and build on top of that, possibly creating a short chain of doomed invalid blocks that confirm an invalid spend. This can confuse SPV wallets, who might see multiple confirmations of a transaction and accept the funds, but later find that in fact it is invalid under the now-activated softfork rules.
- Thus, a very high threshold was imposed. 95% is considered safe. 50% is definitely not safe. Due to variance in the mining process, 80% could also be potentially unsafe (i.e. 80% of blocks signaling might have a good chance of coming from only 60% of miners), so a threshold of 95% was considered "safe enough for Bitcoin work".
- Why have a timeout that disables the upgrade?
- Before BIP9, what was used was either flag day or BIP34. BIP34 had no flag day of activation or a bit, instead, it was just a 95% threshold to signal an nVersion value greater than a specific value. Actually, it was two thresholds: at 75%, blocks with the new nVersion would have the new softfork rules imposed, but at 95% blocks with the old nVersion would be rejected (and only the new blocks, with the new softfork rules, were accepted). For one, between 75% and 95%, there was a situation where the softfork was only "partially imposed", only blocks signaling the new rules would actually have those rules, but blocks with the old rules were still valid. This was fine for BIP34, which only added rules for miners with negligible use for non-miners.
- The same activation process for BIP34 was used for BIP66. After BIP66 reached 95%, however, a single miner mined an invalid-for-BIP66 block that still signalled BIP66 support. It turned out that of the 95% signaling BIP66 support, only about 50% were actually imposing the BIP66 new rules. The rest signalled support without upgrading their software to support new rules. This lead to many chainsplits and chaos with SPV nodes.
- The reasons miners signalled support was because they felt they were being pressured to signal support. So they signalled support, with plans to actually upgrade later, but because of the widespread signalling, the new BIP66 version locked in before upgrade plans were finished. Thus, the timeout that disables the upgrade was added in BIP9 to allow miners an escape hatch.
The Great Battles of the SegWit Wars
SegWit not only fixed transaction malleability, it also created a practical softforkable blocksize increase that also rebalanced weights so that the cost of spending a UTXO is about the same as the cost of creating UTXOs (and spending UTXOs is "better" since it limits the size of the UTXO set that every fullnode has to maintain).
So SegWit was written, the activation was decided to be BIP9, and then.... miner signalling stalled at below 75%.
Thus were the Great SegWit Wars started.
BIP9 Feature Hostage
If you are a miner with at least 5% global hashpower, you can hold a BIP9-activated softfork hostage.
You might even secretly want
the softfork to actually push through. But you might want to extract concession from the users and the developers. Like removing the halvening. Or raising or even removing the block size caps (which helps larger miners more than smaller miners, making it easier to become a bigger fish that eats all the smaller fishes). Or whatever.
With BIP9, you can
hold the softfork hostage. You just hold out and refuse to signal. You tell everyone you will signal, if and only if certain concessions are given to you.
This ability by miners to hold a feature hostage was enabled because of the miner-exit allowed by the timeout
on BIP9. Prior to that, miners were considered little more than expendable security guards, paid for the risk they take to secure the network, but not special in the grand scheme of Bitcoin.
ASICBoost was a novel way of optimizing SHA256 mining, by taking advantage of the structure of the 80-byte header that is hashed in order to perform proof-of-work. The details of ASICBoost are out-of-scope here but you can read about it elsewhere
Here is a short summary of the two
types of ASICBoost, relevant to the activation discussion.
- Overt ASICBoost - Manipulates the unused bits in nVersion to reduce power consumption in mining.
- Covert ASICBoost - Manipulates the order of transactions in the block to reduce power consumption in mining.
Now, "overt" means "obvious", while "covert" means hidden. Overt ASICBoost is obvious because nVersion bits that are not currently in use for BIP9 activations are usually 0 by default, so setting those bits to 1 makes it obvious that you are doing something weird (namely, Overt ASICBoost). Covert ASICBoost is non-obvious because the order of transactions in a block are up to the miner anyway, so the miner rearranging the transactions in order to get lower power consumption is not going to be detected.
Unfortunately, while Overt ASICBoost was compatible with SegWit, Covert ASICBoost was not
. This is because, pre-SegWit, only the block header Merkle tree committed to the transaction ordering. However, with SegWit, another Merkle tree exists, which commits to transaction ordering as well. Covert ASICBoost would require more computation to manipulate two Merkle trees, obviating the power benefits of Covert ASICBoost anyway.
Now, miners want to use ASICBoost (indeed, about 60->70% of current miners probably use the Overt ASICBoost nowadays; if you have a Bitcoin fullnode running you will see the logs with lots of "60 of last 100 blocks had unexpected versions" which is exactly what you would see with the nVersion manipulation that Overt ASICBoost does). But remember: ASICBoost was, at around the time, a novel
improvement. Not all miners had ASICBoost hardware. Those who did, did not want it known that they had ASICBoost hardware, and wanted to do Covert ASICBoost!
But Covert ASICBoost is incompatible with SegWit, because SegWit actually has two Merkle trees of transaction data, and Covert ASICBoost works by fudging around with transaction ordering in a block, and recomputing two Merkle Trees is more expensive than recomputing just one (and loses the ASICBoost advantage).
Of course, those miners that wanted Covert ASICBoost did not want to openly admit
that they had ASICBoost hardware, they wanted to keep their advantage secret because miners are strongly competitive in a very tight market. And doing ASICBoost Covertly was just the ticket, but they could not work post-SegWit.
Fortunately, due to the BIP9 activation process, they could hold SegWit hostage while covertly taking advantage of Covert ASICBoost!
UASF: BIP148 and BIP8
When the incompatibility between Covert ASICBoost and SegWit was realized, still, activation of SegWit stalled, and miners were still not openly claiming that ASICBoost was related to non-activation of SegWit.
Eventually, a new proposal was created: BIP148
. With this rule, 3 months before the end of the SegWit timeout
, nodes would reject blocks that did not
signal SegWit. Thus, 3 months before SegWit timeout
, BIP148 would force activation of SegWit.
This proposal was not accepted by Bitcoin Core, due to the shortening of the timeout (it effectively times out 3 months before the initial SegWit timeout). Instead, a fork of Bitcoin Core was created which added the patch to comply with BIP148. This was claimed as a User Activated Soft Fork, UASF, since users could freely download the alternate fork rather than sticking with the developers of Bitcoin Core.
Now, BIP148 effectively is just a BIP9 activation, except at its (earlier) timeout, the new rules would be activated anyway (instead of the BIP9-mandated behavior that the upgrade is cancelled at the end of the timeout
BIP148 was actually inspired by the BIP8
proposal (the link here is a historical version; BIP8 has been updated recently, precisely in preparation for Taproot activation). BIP8 is basically BIP9, but at the end of timeout
, the softfork is activated anyway rather than cancelled.
This removed the ability of miners to hold the softfork hostage. At best, they can delay the activation, but not stop it entirely by holding out as in BIP9.
Of course, this implies risk that not all miners have upgraded before activation, leading to possible losses for SPV users, as well as again re-pressuring miners to signal activation, possibly without the miners actually upgrading their software to properly impose the new softfork rules.
BIP91, SegWit2X, and The Aftermath
BIP148 inspired countermeasures, possibly from the Covert ASiCBoost miners, possibly from concerned users who wanted to offer concessions to miners. To this day, the common name for BIP148 - UASF - remains an emotionally-charged rallying cry for parts of the Bitcoin community.
One of these was SegWit2X. This was brokered in a deal between some Bitcoin personalities at a conference in New York, and thus part of the so-called "New York Agreement" or NYA, another emotionally-charged acronym.
The text of the NYA was basically:
- Set up a new activation threshold at 80% signalled at bit 4 (vs bit 1 for SegWit).
- When this 80% signalling was reached, miners would require that bit 1 for SegWit be signalled to achive the 95% activation needed for SegWit.
- If the bit 4 signalling reached 80%, increase the block weight limit from the SegWit 4000000 to the SegWit2X 8000000, 6 months after bit 1 activation.
The first item above was coded in BIP91
Unfortunately, if you read the BIP91, independently
of NYA, you might come to the conclusion that BIP91 was only about lowering the threshold to 80%. In particular, BIP91 never mentions anything about the second point above, it never mentions that bit 4 80% threshold would also
signal for a later hardfork increase in weight limit.
Because of this, even though there are claims that NYA (SegWit2X) reached 80% dominance, a close reading of BIP91 shows that the 80% dominance was only for SegWit activation, without necessarily a later 2x capacity hardfork (SegWit2X).
This ambiguity of bit 4 (NYA says it includes a 2x capacity hardfork, BIP91 says it does not) has continued to be a thorn in blocksize debates later. Economically speaking, Bitcoin futures between SegWit and SegWit2X showed strong economic dominance in favor of SegWit (SegWit2X futures were traded at a fraction in value of SegWit futures: I personally made a tidy but small amount of money betting against SegWit2X in the futures market), so suggesting that NYA achieved 80% dominance even in mining is laughable, but the NYA text that ties bit 4 to SegWit2X still exists.
Historically, BIP91 triggered which caused SegWit to activate before the BIP148 shorter timeout. BIP148 proponents continue to hold this day that it was the BIP148 shorter timeout and no-compromises-activate-on-August-1 that made miners flock to BIP91 as a face-saving tactic that actually removed
the second clause of NYA. NYA supporters keep pointing to the bit 4 text in the NYA and the historical activation of BIP91 as a failed promise by Bitcoin developers.
Taproot Activation Proposals
There are two primary proposals I can see for Taproot activation
- Modern Softfork Activation.
We have discussed BIP8: roughly, it has bit
, if 95% of miners signal bit
it activates, at the end of timeout
it activates. (EDIT: BIP8 has had recent updates: at the end of timeout
it can now activate or fail. For the most part, in the below text "BIP8", means BIP8-and-activate-at-timeout, and "BIP9" means BIP8-and-fail-at-timeout)
So let's take a look at Modern Softfork Activation!
Modern Softfork Activation
This is a more complex activation method, composed of BIP9 and BIP8 as supcomponents.
- First have a 12-month BIP9 (fail at timeout).
- If the above fails to activate, have a 6-month discussion period during which users and developers and miners discuss whether to continue to step 3.
- Have a 24-month BIP8 (activate at timeout).
The total above is 42 months, if you are counting: 3.5 years worst-case activation.
The logic here is that if there are no problems, BIP9 will work just fine anyway. And if there are problems, the 6-month period should weed it out. Finally, miners cannot hold the feature hostage since the 24-month BIP8 period will exist anyway.
PSA: Being Resilient to Upgrades
Software is very birttle.
Anyone who has been using software for a long time has experienced something like this:
- You hear a new version of your favorite software has a nice new feature.
- Excited, you install the new version.
- You find that the new version has subtle incompatibilities with your current workflow.
- You are sad and downgrade to the older version.
- You find out that the new version has changed your files in incompatible ways that the old version cannot work with anymore.
- You tearfully reinstall the newer version and figure out how to get your lost productivity now that you have to adapt to a new workflow
If you are a technically-competent user, you might codify your workflow into a bunch of programs. And then you upgrade one of the external pieces of software you are using, and find that it has a subtle incompatibility with your current workflow which is based on a bunch of simple programs you wrote yourself. And if those simple programs are used as the basis of some important production system, you hve just screwed up because you upgraded software on an important production system.
And well, one of the issues with new softfork activation is that if not enough people (users and miners) upgrade to the newest Bitcoin software, the security of the new softfork rules are at risk.
Upgrading software of any kind is always a risk, and the more software you build on top of the software-being-upgraded, the greater you risk your tower of software collapsing while you change its foundations.
So if you have some complex Bitcoin-manipulating system with Bitcoin somewhere at the foundations, consider running two Bitcoin nodes:
- One is a "stable-version" Bitcoin node. Once it has synced, set it up to connect=x.x.x.x to the second node below (so that your ISP bandwidth is only spent on the second node). Use this node to run all your software: it's a stable version that you don't change for long periods of time. Enable txiindex, disable pruning, whatever your software needs.
- The other is an "always-up-to-date" Bitcoin Node. Keep its stoarge down with pruning (initially sync it off the "stable-version" node). You can't use blocksonly if your "stable-version" node needs to send transactions, but otherwise this "always-up-to-date" Bitcoin node can be kept as a low-resource node, so you can run both nodes in the same machine.
When a new Bitcoin version comes up, you just upgrade the "always-up-to-date" Bitcoin node. This protects you if a future softfork activates, you will only receive valid Bitcoin blocks and transactions. Since this node has nothing running on top of it, it is just a special peer of the "stable-version" node, any software incompatibilities with your system software do not exist.
Your "stable-version" Bitcoin node remains the same version until you are ready to actually upgrade this node and are prepared to rewrite most of the software you have running on top of it due to version compatibility problems.
When upgrading the "always-up-to-date", you can bring it down safely and then start it later. Your "stable-version" wil keep running, disconnected from the network, but otherwise still available for whatever queries. You do need some system to stop the "always-up-to-date" node if for any reason the "stable-version" goes down (otherwisee if the "always-up-to-date" advances its pruning window past what your "stable-version" has, the "stable-version" cannot sync afterwards), but if you are technically competent enough that you need
to do this, you are technically competent enough to write such a trivial monitor program (EDIT: gmax notes
you can adjust the pruning window by RPC commands to help with this as well).
This recommendation is from gmaxwell
on IRC, by the way.
Bitcoin Knots began in 2011 December as Bitcoin Next-Test, a derivative of Bitcoin Core's "master" branch with upcoming and proposed features merged in manner as unbiased as possible toward the changes themselves. In 2014, due to time constraints and a desire to maintain a more stable and well-maintained derivative, I began instead to base the branch off the latest stable release, and include ... bitcoind Systemd Unit service (with templates). GitHub Gist: instantly share code, notes, and snippets. Bitcoin Core integration/staging tree. Contribute to bitcoin/bitcoin development by creating an account on GitHub. Bitcoin Core 0.10 implements BIP 66, which introduces block version 3, and a new consensus rule, which prohibits non-DER signatures. Such transactions have been non-standard since Bitcoin v0.8.0 (released in February 2013), but were technically still permitted inside blocks. This change breaks the dependency on OpenSSL’s signature parsing, and is required if implementations would want to ... Follow the steps below to enable prune mode in Bitcoin core wallet. 1. First locate your wallet configuration file. If you don’t find bitcoin.conf file in your Bitcoin directory then create one. 2. Open wallet, go to settings >> options and click on open configuration file. 3.The config file will be mostly empty. Now close your wallet and in ...
asmr btc crypto sync Bitcoin Blockchain Full Node Sync This guide will show you how to recover your lost or missing bitcoins from the bitcoin core wallet while it's still syncing. My Book: https://www.amazon.com/... How to Download & Verify the Bitcoin Core Wallet - Duration: 15:29. Rex Kneisley 18,951 views. 15:29. John McAfee: The Only Way To Secure Your Bitcoins (The Cryptoverse #141) ... Installed Bitcoin on my network. Helping the cause. Will be mining for BTCs. Binance СЕО LIVE: Bitcoin price prediction & Givе Awaу BTC Binance 3,674 watching Live now Hard Forks Killing Bitcoin, $100,000 TRON Bounty And Ethereum Passes Bitcoin - Duration: 28:07.