uncompiled.com

The last six months have been exciting for Digg’s engineering team. We’re working on a soup-to-nuts rewrite. Not only are we rewriting all our application code, but we’re also rolling out a new client and server architecture. And if that doesn’t sound like a big enough challenge, we’re replacing most of our infrastructure components and moving away from LAMP.

Perhaps our most significant infrastructure change is abandoning MySQL in favor of a NoSQL alternative. To someone like me who’s been building systems almost exclusively on relational databases for almost 20 years, this feels like a bold move.
What’s Wrong with MySQL?

Our primary motivation for moving away from MySQL is the increasing difficulty of building a high performance, write intensive, application on a data set that is growing quickly, with no end in sight. This growth has forced us into horizontal and vertical partitioning strategies that have eliminated most of the value of a relational database, while still incurring all the overhead.

Relational database technology can be a blunt instrument and we’re motivated to find a tool that matches our specific needs closely. Our domain area, news, doesn’t exact strict consistency requirements, so (according to Brewer’s theorem) relaxing this allows gains in availability and partition tolerance (i.e. operations completing, even in degraded system states). We’re confident that our engineers can implement application level consistency controls much more efficiently than MySQL does generically.

As our system grows, it’s important for us to span multiple data centers for redundancy and network performance and to add capacity or replace failed nodes with no downtime. We plan to continue using commodity hardware, and to continue assuming that it will fail regularly. All of this is increasingly difficult with MySQL.
Choosing an Alternative

Digg is committed to the use and development of open source software and we’re keen to avoid the cost of proprietary large-scale storage solutions. We were inspired by Google and Amazon’s broad use of their non-relational BigTable and Dynamo systems. We evaluated all the usual open source NoSQL suspects. After considerable debate, we decided to go with Cassandra.

Simplistically, Cassandra is a distributed database with a BigTable data model running on a Dynamo like infrastructure. It is column-oriented and allows for the storage of relatively structured data. It has a fully decentralized model; every node is identical and there is no single point of failure. It’s also extremely fault tolerant; data is replicated to multiple nodes and across data centers. Cassandra is also very elastic; read and write throughput increase linearly as new machines are added.

We experimented on our live site, replacing a relatively high scale MySQL component with a Cassandra alernative. These tests went well. You can read more about these experiments here.
Where We Are

At the time of writing, we’ve reimplemented most of Digg’s functionality using Cassandra as our primary datastore. We’ve supplemented Cassandra-based indexing using full text, relational and graph indexing systems. We’re getting used to dealing with eventual consistency.

We’ve been working on Cassandra itself too. We’ve made massive performance improvements: increased comparitor speed, added better compaction threading, reduced logging overhead, added row-level caching and implemented multi-get capability. We’ve also implemented native atomic counters using Zookeeper (you can probably guess why were motivated to add that feature :)

We’ve tested and improved the operational capabilities of Cassandra, upgrading its Rackaware capability, added slow query logging, improved the bulk import functionality and implemented Scribe support for improved logging. We’ve also done a ton of operational testing.

We’re open sourcing all our work on Cassandra.
What’s Next?

Currently our main focus is getting Digg’s latest release into general availability, but we’ll continue to lead the way in championing Cassandra’s development and adoption.

If you’re interested in joining a world-class team using cutting edge, NoSQL technology at scale, check out http://jobs.digg.com

Take it easy,
John Quinn. VP Engineering. (Digg: doofdoofsf, Twitter: doofdoofsf)

Source

We’ve made two improvements to AWS Import/Export.

You can now send us a “raw” or internal SATA drive all by itself, with no need for an enclosure. You don’t have to send connectors, cables, or power cords. Raw SATA drives appear to be the most cost-effective way to send large amounts of data from place to place.

If you have a SATA cradle (I use this one at home; others have told me that they like this one), you can connect the drive to your desktop machine without having to open up the enclosure.

Also, you can now send us drives with capacities up to 4 TB. Customers with the need to import or export large amounts of data will reduce the number of devices needed.

Don’t forget that tools like Bucket Explorer, the CloudBerry S3 Explorer, and the S3Fox Explorer make it easy to create your Import and Export jobs.

Source

Automobile giant Ford Motor this year will debut vehicles with built-in WiFi — along with enhanced security features to prevent data breaches via its new cars.

Ford has offered the so-called Sync technology service it co-developed with Microsoft in most of its Ford, Lincoln, and Mercury vehicles since 2008. The technology lets drivers run their Bluetooth-enabled mobile phones and digital media players via their vehicles and use voice commands to operate them, for instance.

The automaker announced today that the second generation of its Sync technology — due out later this year and to include a full Windows CE operating system with a new driver interface called MyFordTouch — will come with a built-in browser and secured WiFi access. It will first debut in the 2011 Ford Edge and 2011 MKX Lincoln, and later, in the 2012 Ford Focus.

“We really began to focus on the security side when we began launching Sync, and it was [originally] for working with phones and media players,” says Jim Buczkowski, director of Ford electronics and electrical systems engineering. “Now we’re extending that system connectivity to include WiFi as another data path for customers in their vehicles … and we’re extending that security model for protecting WiFi.”

The WiFi will be broadcast via Sync using a USB-based modem, and Ford has updated its on-board firewalls to protect both the WiFi network as well as the vehicle’s operations. The WiFi network is set by default to WiFi Protected Access 2 (WPA2) encryption for secured access to the wireless network. It also will provide anti-malware protection for the MyFordTouch system.

Sukhwinder Wadhwa, manager of the Sync platform and technologies at Ford, says Ford doesn’t consider security to be an add-on feature. “We work closely with the Ford enterprise IT security [group] to use basically the same guiding principals for security” as they use for the enterprise security, Wadhwa says.

“Any software is first verified by Ford engineers and signed by Ford enteprise servers before it gets installed [in the vehicles],” he says.

Wadhwa says Ford also uses internal ethical hacking teams as well as third-party consultants to test out the security of the Sync features.

“They are proud that they enable WPA2 and a firewall by default on the access point, perform pairing over Bluetooth, and have some arbitrary DRM for preventing swapping hard drives of MP3s. It all sounds like pretty vanilla stuff, anything a decent home network set-up has,” says Nate Lawson, principal with Root Labs.

Wadhwa says Ford isn’t aware of any car-hacking incidents with its vehicles to date. “We do not want to have any incidents in the first place,” he says. “We are connecting consumer-grade devices [in the vehicle], and we want to make sure out of the chute we are protected from any bad devices out there, like memory sticks or whatever they put [into the vehicle],” he says.

Wadhwa says the hardware-based firewall technology is made up of two “separate entities” so that the consumer side of the firewall that handles what can connect can’t pass information to the vehicle’s processor, or vice versa. ”

All of Ford’s vehicles in the next five years will come with the secure WiFi option, according to Ford.

Meanwhile, the automaker’s Sync service, which comes standard in some higher-end models and for an optional monthly fee in other models, already comes with phone-pairing protection, an encrypted jukebox hard drive for the driver’s music library, a valet-mode option that locks all programmed navigation destinations from view, an engine immobilizer, and keyless entry features.

Source

Cisco (NASDAQ: CSCO) today announced a major advancement in Internet networking — the Cisco® CRS-3 Carrier Routing System (CRS) — designed to serve as the foundation of the next-generation Internet and set the pace for the astonishing growth of video transmission, mobile devices and new online services through this decade and beyond.

With more than 12 times the traffic capacity of the nearest competing system, the Cisco CRS-3 is designed to transform the broadband communication and entertainment industry by accelerating the delivery of compelling new experiences for consumers, new revenue opportunities for service providers, and new ways to collaborate in the workplace.

Overview:

* The Cisco CRS-3 triples the capacity of its predecessor, the Cisco CRS-1 Carrier Routing System, with up to 322 Terabits per second, which enables the entire printed collection of the Library of Congress to be downloaded in just over one second; every man, woman and child in China to make a video call, simultaneously; and every motion picture ever created to be streamed in less than four minutes.

* The Cisco CRS-3 enables unified service delivery of Internet and cloud services with service intelligence spanning service provider Internet Protocol Next-Generation Networks (IP NGNs) and data center. The Cisco CRS-3 also provides unprecedented savings with investment protection for the nearly 5,000 Cisco CRS-1 deployed worldwide. Cisco’s cumulative investment in the Cisco CRS family is $1.6 billion, further underscoring the company’s commitment.

* AT&T, one of the world’s largest telecommunications companies, recently tested the Cisco CRS-3 in a successful completion of the world’s first field trial of 100-Gigabit backbone network technology, which took place in AT&T’s live network between New Orleans and Miami. The trial advances AT&T’s development of the next generation of backbone network technology that will support the network requirements for the growing number of advanced services offered by AT&T to consumer and business customers, both fixed and mobile.

* The Cisco CRS-3 is currently in field trials, and its pricing starts at $90,000 U.S.

Highlights and Capabilities for the Next-Generation Internet:

* Unmatched Scale: With a proven multi-chassis architecture, the Cisco CRS-3 can deliver up to 322 tbps of capacity, more than tripling the 92 tbps capacity of the Cisco CRS-1 and representing more than 12 times the capacity of any other core router in the industry.
* Unique Core and Data Center/Cloud Services Intelligence: In addition to capacity requirements, the growths of mobile and video applications are creating new multidirectional traffic patterns with the increasing emergence of the data center cloud. The new Cisco Data Center Services System provides tight linkages between the Cisco CRS-3, Cisco Nexus family and Cisco Unified Computing System (UCS) to enable unified service delivery of cloud services. This intelligence also includes carrier-grade IPv6 (CGv6) and core IP/MPLS technologies that permit new IP NGN architectural efficiencies required to keep pace with the rapidly growing cloud services market. Unique capabilities include:
o Network Positioning System (NPS) — provides Layers 3 to 7 application information for best path to content, improving consumer and business experiences while reducing costs.
o Cloud virtual private network (VPN) for Infrastructure as a Service (IaaS)-enables ‘pay-as-you-go’ for compute, storage and network resources by automating Cisco CRS-3 and Cisco Nexus Inter-Data center connections for Cisco UCS.
* Unprecedented Savings: The Cisco CRS-3 offers dramatic operational expense savings and up to 60 percent savings on power consumption compared to competitive platforms. The Cisco CRS-3 also delivers significant capital expenditures savings and investment protection for existing Cisco CRS-1 customers. The new capabilities in the platform can be achieved by reusing the existing chassis, route processors, fans and power systems with the addition of new line cards and fabric. These upgrades can be performed in-service and be provided by Cisco Services to ensure a smooth transition.
* Silicon Innovation: The Cisco CRS-3 is powered by the new Cisco QuantumFlow Array Processor, which unifies the combined power of six chips to work as one, enabling unprecedented levels of service capabilities and processing power. Making this implementation even more unique is its ability to deliver capabilities with a fraction of the power required by lesser performing chipsets. The Cisco QuantumFlow Array chipset was designed to provide the new system the ability to scale with the ever increasing demands being placed on the IP NGN by the many different applications and billions of devices being used by both businesses and consumers in the Zettabyte era.

Source

Window Snyder, ex-senior security strategist at Microsoft and from late 2006 to late 2008 head of security at the Mozilla Foundation, is joining Apple as senior security product manager. Snyder’s experience could help Apple deal with security problems on a range of fronts.

Back in mid 2009, security specialist Rich Mogull called for Apple to pay greater attention to security, including creating a security response team to deal with reports on vulnerabilities in previously released software. Because many Apple projects are based on open source software, it is, according to Mogull, especially important to keep an eye on the security status of these projects and to have a rapid response.

Apple has always tended to be a little behind the game when it comes to closing holes in the software it uses – frequently even in its own in-house WebKit. Major vulnerabilities are regularly found in the Safari web browser. Security services provider TippingPoint’s Zero Day Initiative currently lists seven critical, as-yet-unpublished security vulnerabilities in Apple software, all of which, according to US media reports, are in Safari.

Apple did, however, introduce a range of improvements to its operating system on releasing Snow Leopard and reacted to criticism of its poor security. Whilst the winner of the 2009 Pwn2Own hacker contest describes the hurdles to hacking Mac OS X as low (“Writing exploits for Vista is hard work, writing exploits for Mac is fun”), the introduction of address space layout randomisation (ALSR) and data execution prevention (DEP) will pose greater problems for attackers.

Under Snyder’s stewardship, the Mozilla Foundation began to introduce a better model for measuring the security of its Firefox browser. At Microsoft, she was involved in developing Windows Server 2003 and Windows XP Service Pack 2. She was also responsible for links between consultants and the Microsoft product team and between her company and security companies and researchers.

Source

Subject: DN: www.cryptome.org; Registrar: Network Solutions; Host: Network Solutions – Demand for Immediate Take Down – Notice of Infringing Activity – MS Ref. 304277
Date: Thu, 25 Feb 2010 12:22:59 -0500
From: “DMCA”
To: “John Young”

We would like to notify you that Microsoft has contacted us regarding www.cryptome.org. Microsoft has withdrawn their DMCA complaint. As a result www.cryptome.org has been reactivated and this matter has been closed. Please allow time for the reactivation to propagate throughout the various servers around the world.

Linda L. Larsen, Designated Agent
Network Solutions, LLC
Telephone: 703.668.5615
Facsimile: 703.668.5959
Email: dmca[at]networksolutions.com

_________

Subject: DN: www.cryptome.org; Registrar: Network Solutions; Host: Network Solutions – Demand for Immediate Take Down – Notice of Infringing Activity – MS Ref. 304277
Date: Thu, 25 Feb 2010 13:09 -0500
To: “DMCA”
From: “John Young”

Dear Ms. Larsen,

You may know we are publishing our email exchanges to help readers understand the process. Could you describe means by which Microsoft withdrew their DMCA complaint?

Regards,

John Young

__________

Subject: RE: DN: www.cryptome.org; Registrar: Network Solutions; Host: Network Solutions – Demand for Immediate Take Down – Notice of Infringing Activity – MS Ref. 304277
Date: Thu, 25 Feb 2010 13:24:52 -0500
From: “DMCA”
To: “John Young”

We received an email from Microsoft’s counsel withdrawing the complaint. Accordingly, we restored access and notified you of our action.

If you have any questions, please feel free to call me.

Linda L. Larsen, Designated Agent
Network Solutions, LLC
Telephone: 703.668.5615
Facsimile: 703.668.5959
Email: dmca[at]networksolutions.com

__________

Subject: RE: DN: www.cryptome.org; Registrar: Network Solutions; Host: Network Solutions – Demand for Immediate Take Down – Notice of Infringing Activity – MS Ref. 304277
Date: Thu, 25 Feb 2010 13:42 -0500
To: “DMCA”
From: “John Young”

Could we get a copy of the Microsoft email? For the public record.

Thanks, John

__________

Subject: DN: www.cryptome.org; Registrar: Network Solutions; Host: Network Solutions – Demand for Immediate Take Down – Notice of Infringing Activity – MS Ref. 304277
Date: Thu, 25 Feb 2010 14:09:47 -0500
From: “DMCA”
To: “John Young”

Mr. Young,

Pursuant to your request, attached please find the email correspondence containing Microsoft’s withdrawal of its “takedown request”.

Respectfully,

Linda L. Larsen, Designated Agent
Network Solutions, LLC
Telephone: 703.668.5615
Facsimile: 703.668.5959
Email: dmca[at]networksolutions.com

__________

X-MimeOLE: Produced By Microsoft Exchange V6.5
Received: from opsmail.prod.netsol.com ([10.221.32.60]) by nsiva-exchange4.CORPIT.NSI.NET with Microsoft SMTPSVC(6.0.3790.3959); Wed, 24 Feb 2010 22:47:25 -0500
MIME-Version: 1.0
Content-Type: multipart/alternative;
boundary=”—-_=_NextPart_003_01CAB5CD.3E340480″
Received: from corpcm3 (corpcm3.mgt.netsol.com [10.221.32.102]) by opsmail.prod.netsol.com (8.12.10/8.12.10) with ESMTP id o1P3lOsM023759 for ; Wed, 24 Feb 2010 22:47:24 -0500 (EST)
Received: from [10.253.64.77] ([10.253.64.77:43581] helo=networksolutions.com) by corpcm3 (envelope-from ) (ecelerity 2.2.2.41 r(31179/31189)) with ESMTP id E2/39-15380-3C2F58B4; Wed, 24 Feb 2010 22:47:15 -0500
Received: (qmail 23471 invoked from network); 25 Feb 2010 03:45:41 -0000
Received: from dchost2.cov.com (HELO CBIEXI02DC.cov.com) (216.200.93.137) by tip2.lb.netsol.com with SMTP; 25 Feb 2010 03:45:41 -0000
Received: from cbiexm02sf.cov.com ([172.16.160.88]) by CBIEXI02DC.cov.com with Microsoft SMTPSVC(6.0.3790.3959); Wed, 24 Feb 2010 22:46:57 -0500
Content-class: urn:content-classes:message
Subject: Re: Ticket Number 1-452132847
Date: Wed, 24 Feb 2010 22:46:56 -0500
Message-ID: <54F83DC1AC2D7443AA904FFD32E2DAA40877B437[at]cbiexm02sf.cov.com>
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
Thread-Topic: Re: Ticket Number 1-452132847
Thread-Index: Acq06IFo420gHHQPThWg1v/l3yM7TAAAjYcAAABtzJAAABl6EAAAmcd6ADcZlUA=
From: “Cox, Evan”
To: “DMCA”
Cc: “internet4[at]microsoft-antipiracy.com”

Dear Ms. Larsen:

I am outside counsel to Microsoft Corporation. I am writing to confirm my telephone message left with your nighttime operator at 7:45 PST this evening to withdraw Microsoft’s takedown request with respect to the file available at http://cryptome.org/isp-spy/microsoft-spy.zip which is the subject of the correspondence below.

While Microsoft has a good faith belief that the distribution of the file that was made available at that address infringes Microsoft’s copyrights, it was not Microsoft’s intention that the takedown request result in the disablement of web acess to the entire cryptome.org website on which the file was made available.

Accordingly, on behalf of Microsoft, I am hereby withdrawing the takedown request and asking that Network Solutions restore internet access to http: cryptome.org as soon as possible.

I can be reached at 415-640-5145 if you wish to discuss this request.

Sincerely,

Evan Cox
Counsel to Microsoft Corporation

Source

According to a study by researchers at the University of Oviedo (Universidad de Oviedo) in Spain, the estimated total value of the 2.6.30 Linux kernel, released in June of 2009, is more than €1 billion. Using the kernel development history from version 2.6.11 to 2.6.30, the researchers calculated the costs by using the Constructive Cost Model 81 (COCOMO 81) and taking the average annual salary for a developer in 2006 in the European Union as a parameter. According to EUROSTAT, that was approximately €31,000. The Linux Foundation published a similar study in October of 2008.

The COCOMO algorithm calculates the value of software using a number of specific metrics, specifically the number of lines of code written. The study looks at the estimated annual research and development (R&D) costs of the kernel releases and shows that the annual Linux kernel R&D cost increased significantly in 2008. Between 2005 and 2006, annual R&D was estimated at between €72 to €94 million, however, in 2008, that number rose to more than €228 million.

The two researchers responsible for the study, Jesús García-García and Mª Isabel Alonso de Magdaleno, will be be presenting their findings at the Concord 2010 conference on corporate R&D taking place on the 3rd and 4th of March, 2010 in Seville, Spain.

Source

Cryptome, the whistleblower site that serves as a repository for “documents for publication that are prohibited by governments worldwide, in particular material on freedom of expression, privacy, cryptology, dual-use technologies, national security, intelligence, and secret governance”, has been taken down on Wednesday afternoon by its hosting provider, Network Solutions, which also had the domain “legally locked”.

What it means is that domain information can’t be modified and the domain name transferred – only the registration can be renewed. This action from Network solutions was motivated by a Digital Millennium Copyright Act (DMCA) complaint filed by Microsoft against Cryptonome and its owners, regarding the publication of their Global Criminal Compliance Handbook, a document that reveals to users things that Microsoft would not like to become common knowledge.

In it you can find information about what records are retained and for how long, and what information can and will be given to law enforcement and intelligence agencies if requested by subpoena. Microsoft is not the only company whose “spy guide” has been published by Cryptome, but it’s apparently the one with the most clout.

ReadWriteWeb reports that once the complaint was filed and they requested of Paul Young, one of the owners of Cryptome, to take the document off the website, he refused. So the ISP intervened with a warning that said that if the document wasn’t removed by Thursday, they would disable the site. And so they did – one day before the imposed deadline.

In the complaint, Microsoft states as the reason for their request an infringement of copyright laws. The Electronic Frontier Foundation (EFF), the well-known international digital rights watchdog group, spoke up: “We find it troubling that copyright law is being invoked here. Microsoft doesn’t sell this manual. There’s no market for this work. It’s not a copyright issue. John’s copying of it is fair use. We don’t do this anywhere else in speech law.”

Cryptome has been active since 1996, and this is the first time that someone has succeeded in their mission to shut it down. Young has filed a counter-notification, and we probably won’t have to wait long for the next installment of this story. If Microsoft doesn’t forward a notice of litigation, Network Solutions will reactivate the Website and unlock the domain in no more than 14 business days. In the meantime, the website is temporarily available here.

If you want to read Microsoft’s “spy guide”, you can download it at Wikileaks, who has also offered to to host Cryptome on their multi-jurisdictional network-outside the US.

Source

The web site cryptome.org is currently online at http://cryptomeorg.siteprotect.net/ until the domain can be transferred away from Network Solutions. The following is from the temporary site:

This is temporary Cryptome address until the Cryptome.org domain is transferred. Network Solutions shut Cryptome.org and has placed a “legal lock” on the domain name, preventing its transfer, until the “dispute” is settled. Some recent files are available now and the full collection is being transferred.

Server shipments increased in the fourth quarter of 2009, but revenue fell as x86 servers continued to bite into the declining market for Unix servers with RISC and Itanium chips, Gartner said in a study released on Wednesday.

Worldwide server shipments totaled 2.2 million during last year’s fourth quarter, growing 4.5 percent compared to the fourth quarter of 2008. Worldwide server sales totaled US$12.6 billion during the fourth quarter last year, a 3.2 percent drop from server sales in the same quarter of 2008. Worldwide sales of x86 servers rose 14.3 percent to $7.6 billion during the 2009 fourth quarter, while RISC and Itanium server sales declined by 20 percent to about $3 billion.

“The fourth quarter of 2008 was quite weak, so the fourth quarter of 2009 did not have to produce huge x86 server numbers to result in an increase,” said Jeffrey Hewitt, research vice president at Gartner, in a statement. “At the same time, other segments like RISC/Itanium Unix and mainframes remained constrained and that exerted downward pressure on overall vendor revenue results.”

IBM and Intel in early February released new chips for highly scalable servers that require high uptime. At the time of the Power7 chip launch, the senior vice president of IBM’s Systems and Technology group, Rod Adkins, said the Unix market is a sizeable and healthy market to the tune of $14 billion or $15 billion annually. Adkins also said IBM will try to provide more competitive pricing options on its Power7 servers to take on the “traditional” server market.

Industry analysts went on to say that the new chips will have little effect on reviving the declining sales of scalable servers running on the Unix OS, analysts said. The x86 servers are cheaper, are getting more powerful and have a more readily available software stack.

The x86 server market was trending toward a larger use of blades, Hewitt said. Blade server shipments grew by 11.1 percent during the quarter, while revenue grew by 22.1 percent.

IBM was the top server vendor during the fourth quarter, with sales of $4.1 billion, a drop of 5.9 percent year over year and a 32.7 percent market share. In a close second was Hewlett-Packard, with server sales of $3.95 billion, recording a small growth of 0.4 percent and 31.3 percent market share. Dell’s server revenue increased by 8.3 percent to $1.5 billion, putting it in third place. Sun Microsystems, which has since been acquired by Oracle, saw a massive 23.5 percent drop in server sales.

IBM was also first in Unix servers with a 40.5 percent market share during the fourth quarter of 2009. The company registered sales of $1.2 billion, a year-over-year drop of 11.1 percent. Hewlett-Packard was second, with sales of $876 million, a year-over-year drop of 19.9 percent. Sun shipped the largest number of Unix servers during the fourth quarter, but was third in revenue with sales of $753.8 million, a year-over-year drop of 29.1 percent. Fujitsu was fourth, recording the largest drop of 55.4 percent.

Hewlett-Packard topped the x86 server market with sales of $2.8 billion, a year-over-year growth rate of 15.7 percent, and a market share of 38 percent. In a distant second was Dell, recording an 8.3 percent year-over-year sales growth to reach $1.5 billion, for 20 percent market share. IBM was in third place, with its server sales growing 37 percent to reach $1.4 billion. Sun and Fujitsu were in fourth and fifth places, respectively.

Source