Data Engineering

All things related to data engineering - databases, distributed systems, big data and much more

Sources

Updated at
Tue Jan 18 20:11:59

	How to develop a Gateway between Datalake and the BI tools? Does ot possible with custom JDBC which allows data profiling and data selection before push them to the BI tool? by /u/kgsami 11 minutes ago.
	Apache Beam pipelines by /u/MrBreadboard 35 minutes ago.
	Experience with Anomalo or other data testing solutions? by /u/magnimeelcul 1 hour ago.
	Is ab initio experience valuable? by /u/signacaste 1 hour ago.
	SQL data design books or resources by /u/outandaboutbc 2 hours ago.
	Delta Lake learning resources recommendations? by /u/No_Engine1637 2 hours ago.
	Data Architect vs Data Engineer by /u/authentichooman 2 hours ago.
	Dagster - are these people from the future of data engineering? Others' opinions? by /u/britishbanana 3 hours ago.
	Measuring Data Warehousing/Data Lake ROI by /u/Dpohl1nthaho1e 4 hours ago.
	Fool-Proof Formula to Extracting Strings With SQL by Madison Schott 4 hours ago.
	Azure architect interview questions by /u/dragonachu117 4 hours ago.
	Intermediate SQL for Everyone by Matt Sosna 4 hours ago.
	Data engineering guidance by /u/adaptrix 5 hours ago.
	8 Essential Python Techniques for Data Engineers and Analysts (with code samples) by Cameron Warren 5 hours ago.
	Private vs public cloud by /u/Minimum-Membership-8 6 hours ago.
	Why is a DW needed? by /u/stigmatic666 7 hours ago.
	Houskeeping in Data Vault 2.0 by /u/marco0112 8 hours ago.
	Airflow loops vs dynamic # of tasks by /u/mrflamingosaurus 10 hours ago.
	Thoughts about this curriculum? Can I qualify as a good DE? How relevant are these tools in the current market trend? by /u/Unable-Anywhere9666 11 hours ago.
	[Tutorial] Faster Optimization using GPU Multiprocessing by /u/yamqwe 12 hours ago.
	Interview question about Single Source of Truth? by /u/kirkiter 15 hours ago.
	How to grow at a small company? by /u/z0234 20 hours ago.
	How will you design the validation layer between Kafka and Vertica? by /u/redditthrowaway0315 23 hours ago.
	Tracking user activity with dbt by Suela Isaj yesterday
	Automated Data Quality Management Through Machine Learning With Anomalo - Episode 256 by tmacey 2 days ago.
	An Introduction To Data And Analytics Engineering For Non-Programmers - Episode 255 by tmacey 3 days ago.
	Google Cloud Storage (GCS) to BigQuery the simple way by Jim Barlow 4 days ago.
	18 - Multi-Version Concurrency Control (CMU Intro to Database Systems / Fall 2021) by CMU Database Group 4 days ago.
	Fluree - Cloud-Native Ledger Graph Database (Brian Platz) by CMU Database Group 5 days ago.
	The Pinecone Vector Database System (Edo Liberty) by CMU Database Group 5 days ago.
	Convex: Life Without a Backend Team (James Cowling) by CMU Database Group 6 days ago.
	How Clearcover Increased Quality Coverage for ELT by 70 Percent by Barr Moses 6 days ago.
	The Future of the Modern Data Stack in 2022 by Prukalpa 7 days ago.
	How to Build Powerful Airflow DAGs for Big Data Workflows in Python by Richard Pelgrim 7 days ago.
	20 - Database Recovery (CMU Intro to Database Systems / Fall 2021) by CMU Database Group 7 days ago.
	How To Connect To A Snowflake Database With Python by AnBento 8 days ago.
	21 - Introduction to Distributed Databases (CMU Intro to Database Systems / Fall 2021) by CMU Database Group 9 days ago.
	Open Source Reverse ETL For Everyone With Grouparoo - Episode 254 by tmacey 10 days ago.
	Data Observability Out Of The Box With Metaplane - Episode 253 by tmacey 10 days ago.
	17 - Timestamp Ordering Concurrency Control (CMU Intro to Database Systems / Fall 2021) by CMU Database Group 12 days ago.
	Creating Shared Context For Your Data Warehouse With A Controlled Vocabulary - Episode 252 by tmacey 16 days ago.
	A Reflection On The Data Ecosystem For The Year 2021 - Episode 251 by tmacey 16 days ago.
	19 - Database Logging (CMU Intro to Database Systems / Fall 2021) by CMU Database Group 16 days ago.
	Query Optimization and Acceleration at Dremio (Steven Phillips + Vivekanand Vellanki) by CMU Database Group 17 days ago.
	Revisiting The Technical And Social Benefits Of The Data Mesh - Episode 250 by tmacey 22 days ago.
	Exploring The Evolving Role Of Data Engineers - Episode 249 by tmacey 22 days ago.
	How We Build Firebolt (Benjamin Wagner) by CMU Database Group 28 days ago.
	Fast And Flexible Headless Data Analytics With Cube.JS - Episode 248 by tmacey 28 days ago.
	Building A System Of Record For Your Organization's Data Ecosystem At Metaphor - Episode 247 by tmacey 29 days ago.
	25 - Final Review (CMU Intro to Database Systems / Fall 2021) by CMU Database Group 29 days ago.
	Apache Arrow: High-Performance Columnar Data Framework (Wes McKinney) by CMU Database Group 41 days ago.
	23 - Distributed OLAP Databases (CMU Intro to Database Systems / Fall 2021) by CMU Database Group 56 days ago.
	22 - Distributed OLTP Databases (CMU Intro to Database Systems / Fall 2021) by CMU Database Group 60 days ago.
	Vertica – High Performance Over Varying Terrain (Stephen Walkauskas) by CMU Database Group 67 days ago.
	Improving HDFS I/O Utilization for Efficiency by Kevin Cheng 97 days ago.
	Real-Time Exactly-Once Ad Event Processing with Apache Flink, Kafka, and Pinot by Jacob Tsafatinos 117 days ago.
	YAML Generator for Funnel YAML Files: Streamlining the Mobile Data Workflow Process by Jack Ning 124 days ago.
	Streaming Real-Time Analytics with Redis, AWS Fargate, and Dash Framework by Piyush Choudhary 138 days ago.
	Enabling Seamless Kafka Async Queuing with Consumer Proxy by Yang Yang 140 days ago.
	How Data Shapes the Uber Rider App by Divya Babu Ravichandran 145 days ago.
	Building Scalable Streaming Pipelines for Near Real-Time Features by Feng Xu 147 days ago.
	Efficiently Managing the Supply and Demand on Uber’s Big Data Platform by Zheng Shao 158 days ago.
	Cost-Efficient Open Source Big Data Platform at Uber by Zheng Shao 160 days ago.
	Challenges and Opportunities to Dramatically Reduce the Cost of Uber’s Big Data by Zheng Shao 162 days ago.
	How Uber Achieves Operational Excellence in the Data Quality Experience by Ying Zou 166 days ago.
	Containerizing Apache Hadoop Infrastructure at Uber by Mithun Mathew 180 days ago.
	Analyzing Customer Issues to Improve User Experience by Nimesh Agarwal 186 days ago.
	Automating Merchant Live Monitoring with Real-Time Analytics: Charon by Marco Vita 264 days ago.
	My thoughts on the data mesh by Daniel Abadi 299 days ago.
	Uber’s Journey Toward Better Data Culture From First Principles by Krishna Puttaswamy 308 days ago.
	Turning Metadata Into Insights with Databook by Sunheng Taing 434 days ago.
	Operating Apache Pinot @ Uber Scale by Yupeng Fu 455 days ago.
	Inside Uber ATG’s Data Mining Operation: Identifying Real Road Scenarios at Scale for Machine Learning by Steffon Davis 595 days ago.
	Monitoring Data Quality at Scale with Statistical Modeling by Ye Henry Li 621 days ago.
	Building a Backtesting Service to Measure Model Performance at Uber-scale by Sam Xiao 705 days ago.
	Women in Data Science at Uber: Moving the World With Data in 2020—and Beyond by Emily Bailey 721 days ago.
	Designing a Production-Ready Kappa Architecture for Timely Data Stream Processing by Amey Chaugule 726 days ago.
	Engineering SQL Support on Apache Pinot at Uber by Haibo Wang 734 days ago.
	Uber Visualization Highlights: Displaying City Street Speed Clusters with SpeedsUp by Bryant Luong 747 days ago.
	It's time to rethink how we share data on the Web by Daniel Abadi 761 days ago.
	Uber’s Data Platform in 2019: Transforming Information to Intelligence by Nikhil Joshi 763 days ago.
	Productionizing Distributed XGBoost to Train Deep Tree Models with Large Data Sets at Uber by Joseph Wang 770 days ago.
	Introducing SLOG: Cheating the low-latency vs. strict serializability tradeoff by Daniel Abadi 774 days ago.
	The dangers of conditional consistency guarantees by Daniel Abadi 774 days ago.
	Introduction to Transaction Isolation Levels by Daniel Abadi 774 days ago.
	Correctness Anomalies Under Serializable Isolation by Daniel Abadi 774 days ago.
	Overview of Consistency Levels in Database Systems by Daniel Abadi 774 days ago.
	An explanation of the difference between Isolation levels vs. Consistency levels by Daniel Abadi 774 days ago.
	Partitioned consensus and its impact on Spanner’s latency by Daniel Abadi 775 days ago.
	It’s Time to Move on from Two Phase Commit by Daniel Abadi 775 days ago.
	NewSQL database systems are failing to guarantee consistency, and I blame Spanner by Daniel Abadi 775 days ago.
	Uber Visualization Highlights: How Urban Symphony Adds an Audio Dimension to Visualization by Jon Sadka 790 days ago.
	Taking City Visualization into the Third Dimension with Point Clouds, 3D Tiles, and deck.gl by Ib Green 812 days ago.
	Building a Better Big Data Architecture: Meet Uber’s Presto Team by Wayne Cunningham 832 days ago.
	Science at Uber: Making a Real-world Impact with Data Science by Molly Vorwerck 882 days ago.
	Less is More: Engineering Data Warehouse Efficiency with Minimalist Design by Ritesh Agrawal 888 days ago.
	Science at Uber: Powering Uber’s Ridesharing Technologies Through Mapping by Molly Vorwerck 896 days ago.
	Science at Uber: Bringing Research to the Roads by Molly Vorwerck 903 days ago.
	Science at Uber: Building a Data Science Platform at Uber by Wayne Cunningham 910 days ago.
	Making Apache Spark Effortless for All of Uber by Abhishek Modi 916 days ago.
	Visualizing City Cores with H3, Uber’s Open Source Geospatial Indexing System by Molly Vorwerck 925 days ago.
	Gaining Insights in a Simulated Marketplace with Machine Learning at Uber by Haoyang Chen 939 days ago.
	Using Causal Inference to Improve the Uber User Experience by Totte Harinen 944 days ago.
	Power On: Accelerating Uber’s Self-Driving Vehicle Development with Data by Steffon Davis 959 days ago.
	Second Uber Science Symposium: Exploring Advances in Behavioral Science by Laura Libby 974 days ago.
	An analysis of the strengths and weaknesses of Apache Arrow by Daniel Abadi 1393 days ago.
	Apache Arrow vs. Parquet and ORC: Do we really need a third Apache project for columnar data representation? by Daniel Abadi 1540 days ago.
	Hazelcast and the Mythical PA/EC System by Daniel Abadi 1562 days ago.
	Scala Days 2017 - Jepsen Keynote by Aphyr 1721 days ago.
	Distributed consistency at scale: Spanner vs. Calvin by Daniel Abadi 1748 days ago.
	Jepsen: CockroachDB beta-20160829 by Aphyr 1797 days ago.
	Jepsen: MongoDB 3.4.0-rc3 by Aphyr 1806 days ago.
	IEEE Computer issue on the CAP Theorem by Daniel Abadi 1982 days ago.
	Jepsen: VoltDB 6.3 by Aphyr 2016 days ago.
	Jepsen: Crate 0.54.9 version divergence by Aphyr 2029 days ago.
	Jepsen: RethinkDB 2.2.3 reconfiguration by Aphyr 2187 days ago.
	Jepsen: RethinkDB 2.1.5 by Aphyr 2206 days ago.
	Why MongoDB, Cassandra, HBase, DynamoDB, and Riak will only let you perform transactions on a single data item by Daniel Abadi 2272 days ago.
	Jepsen: Percona XtraDB Cluster by Aphyr 2327 days ago.
	Jepsen: MariaDB Galera Cluster by Aphyr 2330 days ago.
	Jepsen: Chronos by Aphyr 2352 days ago.
	Jepsen: Aerospike by Aphyr 2450 days ago.
	Jepsen: Elasticsearch 1.5.0 by Aphyr 2457 days ago.
	Jepsen: MongoDB stale reads by Aphyr 2464 days ago.
	Jepsen: Elasticsearch by Aphyr 2773 days ago.
	Jepsen: etcd and Consul by Aphyr 2780 days ago.
	Jepsen: RabbitMQ by Aphyr 2783 days ago.
	Computational techniques in Knossos by Aphyr 2802 days ago.
	Strong consistency models by Aphyr 2805 days ago.
	Jepsen: Redis redux by Aphyr 2960 days ago.
	Jepsen: Strangeloop Hangout by Aphyr 3034 days ago.
	Jepsen: RICON East talk by Aphyr 3034 days ago.
	Jepsen: Cassandra by Aphyr 3037 days ago.
	Jepsen: Kafka by Aphyr 3038 days ago.
	Jepsen: NuoDB by Aphyr 3039 days ago.
	Jepsen: Zookeeper by Aphyr 3039 days ago.
	Automating Jepsen by Aphyr 3125 days ago.
	The network is reliable by Aphyr 3151 days ago.
	Asynchronous replication with failover by Aphyr 3164 days ago.
	Jepsen: final thoughts by Aphyr 3165 days ago.
	Jepsen: Riak by Aphyr 3166 days ago.
	Jepsen: MongoDB by Aphyr 3166 days ago.
	Jepsen: On the perils of network partitions by Aphyr 3167 days ago.
	Jepsen: Redis by Aphyr 3167 days ago.
	Jepsen: Postgres by Aphyr 3167 days ago.
	Is Upstart the right way to get college student start-ups funded? by Daniel Abadi 3359 days ago.
	Defending Matt Welsh’s 'Startup University' Post by Daniel Abadi 3493 days ago.
	If all these new DBMS technologies are so scalable, why are Oracle and DB2 still on top of TPC-C? A roadmap to end their dominance. by Daniel Abadi 3534 days ago.
	Replication and the latency-consistency tradeoff by Daniel Abadi 3694 days ago.
	Overview of the Oracle NoSQL Database by Daniel Abadi 3756 days ago.
	Hadoop's tremendous inefficiency on graph data management (and how to avoid it) by Daniel Abadi 3826 days ago.
	Why Sam Madden is wrong about peer review by Daniel Abadi 3897 days ago.
	Why I'm doing a start-up pre-tenure by Daniel Abadi 3948 days ago.

Data Engineering

All things related to data engineering - databases, distributed systems, big data and much more

Sources

Updated at
Tue Jan 18 20:11:59