Author Archives: Ian

CHINA STUFF: Email that works…

via Dangerous Prototypes

mail-dp

Working from China is a constant reminder that the location of internet services is super important. North America and Europe are just a few internet hops from our servers in Hetzner’s carbon neutral, green energy data center in Germany. From China, however, we’re often routed out through Beijing to San Jose, then to New York, London, Amsterdam, and finally Frankfurt. On a good day we can do that with only 2-3% packet loss.

Back in the good old days Gmail’s imap service worked perfectly even though Google was blocked, but eventually email was blocked too. At that point we added a managed mail server at Hetzner in Germany. Despite having our own non-blocked mail server, Hetzner’s data centers are unreachable during peak periods of the day because of peering agreements, congestion, geolocation, and a million other factors not worth arguing about.

The image above is a WinMTR report, a combination of ping and traceroute that helps locate network problems. 219.* is where we exit on China Unicom’s cable in Beijing and connect at San Jose, California. Nearly 50% packet loss, average ping time of 482ms. This is better than normal because San Jose connected directly to Frankfurt when this test was taken, instead of the more typical New York-> London-> Amsterdam route.

So how bad is it exactly? 5 tries to send a simple text email without attachments. “Failed to save draft, try again?” prompts every few seconds. Ability to see mail subject headers, but not download the message text. Want to send or receive an attachment? Better schedule that for late at night or first thing in the morning.

International bandwidth from China

There are three cable landings in China: Beijing, Shanghai, and Guangzhou. Each is dominated by one of the three state-owned ISPs. China Unicom (our provider) is big in the north with international traffic routing through Beijing and around 1TB/s bandwidth. China Telecom is big in the south (e.g. Shenzhen area) with around 2.5TB/s of bandwidth through Guangzhou. There are also a lot of bit players like Great Wall and TopWay that have city and regional backbones that eventually dump onto the major state-owned ISP infrastructure.

Part of our problem is using a northern ISP (Unicom) in the south. All of our traffic is routed to Beijing before leaving China. While we can see Hong Kong from the office and can walk there in about 15 minutes, connections to Hong Kong websites are routed up to Beijing and back for a 3000km+ journey. This isn’t our choice, Unicom has a monopoly in our Huaqiangbei office and nothing else is available.

This was intended to be an epic post using MTR to analyze the optimal routes and geographical locations to stash internet services with the best chance to be reachable from China. That idea bombed because nearly everything changed dramatically day to day. A test from yesterday is different than a test today, which will probably be different than a test tomorrow. Follow below for tests of several major email providers and their accessibility from China.

Rackspace Business Email is a disaster from China

rackspace

Over the past few months we tested a lot of email services. Rackspace is a well respected company with business email hosting for $2 per box per month. Signup from China triggered a review, so we had to call support to complete the order. The Rackspace rep volunteered that they have constant complaints from users in China, not something you want to hear. The MTR report shows why, routing to their imap server is a disaster (30% loss, 361ms average ping).

Microsoft Office365 email works great from China, but is itself a disaster

office365

Probably the most distressing part of daily life in China is using Microsoft’s Bing search engine. It’s a terrible search engine, but it’s always super fast within China. You might even find what you’re looking for if you skip directly to the third page of results, link number 5…

Microsoft’s Office365 imap mail service also works very well inside China, and at $4 per box per month it isn’t very expensive. The MTR report suggests Microsoft is running a server in Hong Kong that connects directly to China Unicom (219.*). The service is fast (21ms ping) and very accessible (very few lost packets).

(Un)fortunately Microsoft’s automatic email migration tool sprayed crap all over the place. Their suggestion was to hire an authorized partner for support. After a while all the subscription plans, conditions, and lack of support started to seem quite sleazy. We canceled when mail migration failed and they demanded 1 year commitments for each test account to help debug it.

ABCHK.net is our email hero

abchk

ABCHK.net is a hosting and email provider located in Hong Kong that specializes in email service for Mainland China. We were super skeptical that we could get a stable connection to Hong Kong, but they provided a test account that blew everyone away. 10 meg attachments? Uploaded and sent in seconds. It works perfectly at all times of the day, and the MTR report shows a direct, clean route to the imap server. No packets to destination lost, 31ms average ping.

For less than $7/month we get five mail boxes and 100GB of storage. Real people answered emails and handled the mail migration from our old server in Germany. After a month we are still extremely thrilled to be able to use email “normally” from inside China. Attachments upload and download super fast, and the server is always 100% reachable.

This applies to Unicom only!

Tests were done on commercial and residential China Unicom 100Mbps fiber connections. China Unicom is not the optimal ISP for Shenzhen though. It makes more sense to be on China Telecom with 2.5x more international bandwidth exiting just an hour north in Guangzhou. In the office Unicom has a monopoly, but we had a Telecom connection installed at home and will run the tests again on Telecom in a few days. Anecdotal evidence from other Telecom users doesn’t seem particularly promising though.

Admittedly this is all niche info, but it was hard won and seemed worth sharing here. At the very least someone in a similar situation might find this on Google. Or on Bing, page 3, result 5.

How to measure laser cut length or time?

via Dangerous Prototypes

800px-Laser_cutting_snowfla

To price laser cutting online we need to extract two pieces of info from uploaded design files. The area of a box surrounding the design, called the bounding box, is the amount of material needed. The length of all the cuts in the design can be used as a rough estimate of the cutting time consumed. Here’s where we’re at…

A dirty cheap laser cut service has been “coming very soon” for far too long. We planned to take SVG files and use a friend’s script to measure the bounding box and cut lengths. Unfortunately we never found a laser cut supplier willing to take low volume/high-mix orders and provide all the pricing info to give instant quotes.

Last week a comment at Hack a Day mentioned that dirty laser cutting is taking forever. Surprised by the interest, we asked around and found some willing suppliers. However, these suppliers only accept dxf, dwg, pdf, eps, and ps formats. SVG won’t do. Now we have a supplier but no way to give instant online quotes.

Conversion

Converting to and from SVG is plagued with wrecked arcs and curves. Potential customer dissatisfaction factor is super high. Absolutely not reliable enough for a commercial service or even for pricing.

Extracting info from CAD programs

Inkscape loads all the above formats, but we were not able to extract total path length from the command line or the shell interface. Autocad.io is an online version of AutoCAD that might be able to provide the design stats and a nice rendering, but if the API is slow or busy are customers supposed to wait minutes for their quotes?

A couple CAD programs seemed promising, but eventually we couldn’t find anything that runs on Linux and can extract the stats we need via the command line.

Measuring scripts

The script that measures PCBs in our store and at DirtyPCBs combines a bit of gerber code knowledge with some math to crank out basic stats without ever loading a CAD program. A small script to extract path length and bounding box size from dxf is very doable after a quick thumb through the spec, however nothing seems to exist online.

The learning curve to implement the dxf format and the debug time to perfect it puts this in the labor of love category. It would be a blast if we really wanted to do it, but since it’s standing in the way of a dirt cheap service that will likely never turn a profit (laser cutting is cheap and common…) there’s no resources to dedicate to writing our own.

Create g-code and simulate the cut

G-code is a popular way to control CNC mills and 3D printers. A CAM processor (yes, like the one in Cadsoft Eagle) combines information about the machine, such as cut speed, with the path in the design file to create a series of cutting steps or instructions. G-code is accepted universally, much like gerber files.

Lots of existing scripts convert dxf/pdf/eps/ps files to g-code. Once the design is in g-code there are several programs that simulate the cut and estimate the cut time (or filament used, etc). This seems like the proper way to do it. As a bonus, it lays the framework to calculate cut time and difficulty for a future DirtyCNC and DirtyInjectionMolding service.

This is as far as we’ve gotten. A dozen Python scripts on github look promising, but nothing directly outputs the info we need without a bit of hacking. We’ll continue to knock together a solution in the next few days.

Please shout out in the comments if you have thoughts on any of these methods or experience with any tool chains that might work. Help us get laser cutting online and we’ll reward you with – what else – free laser cutting service!

Image source: Adam Dingley CC BY SA

Amazon Web Service Beijing woefully inadequate for international use

via Dangerous Prototypes

aws-cn

Amazon Web Service (AWS) is a bunch of server, storage, and other internet resources. AWS datacenters are located around the world so companies can locate websites close to their customers and speed up page load times.

A couple years ago AWS started testing a center in Beijing, China. There’s all sorts of benchmarks on speed within and between AWS datacenters, but the Beijing location is restricted to registered Chinese companies so there’s very little public (English) info on it.

In March the internet in Shenzhen because nearly unusable. While access within China reaches 100Mbps, anything outside China experienced 50%+ packet drops and max speeds of a 20-50Kbps. In a desperate ploy we tried to setup a VPN within China to a better exit point – maybe AWS has a better connection to the outside world than our local ISPs. Keep in mind that this a VPN inside China, the internet is still filtered but is hopefully more consistent and usable.

Signing up for the AWS beta was pretty easy. The account reps were super helpful and spoke perfect English. Everything went smoothly aside from the rep’s utter disbelief that, yes, we are a real Chinese company not a Hong Kong Company, and yes, we don’t want a free trial tier, and yes, we can pay in RMB from a Chinese company bank account. We guess they get a lot of people trying to sign up for free service without a company because they mentioned these things a dozen times.

We setup a “medium instance” with Ubuntu, AWS cult code for a small server without throttling. First things first – apt-get update && apt-get upgrade – to bring the server up to date. Apt pulled from AWS repositories within the same datacenter in China at a shocking 64Kbps! They didn’t even break 100Kbps within their own datacenter! Holy crap. For comparison, we regularly FTP between our servers in Hetzner’s German datacenter at 100Mbps+. Pathetic.

Ok, what about the connection to the outside world? The vast majority of our web traffic exits China from Beijing on one of two cables: San Jose, USA and Tokyo, Japan. Surely we could take advantage of our good speeds within China and then slingshot international traffic at higher speeds through the “superior” AWS infrastructure. Not exactly. A file download from DirtyPCBs.com, located in a super high speed San Jose datacenter, topped out at 16Kbps. Really Amazon?

We did similar experiments with Aliyun, the Chinese version of AWS made by the Alibaba/Aliexpress people. The results were slightly better but still absolutely unusable as a gateway to the internet outside of China.

The quest for an internet solution continues…

HOW-TO: Shipping stuff from China cheap?

via Dangerous Prototypes

shippers

Tyler writes:

What method do you use to ship from China? I am trying to ship 8kg of PLA from a manufacturer in China. I want to figure out how much sea shipping (the cheapest method) would be, but no one is really able to give me a legit answer besides someone like UPS (Yeah $200 is so not happening). I just want to try to get a cheap product from China without blowing the whole cost savings on shipping.

Between DirtyPCBs and Hacker Camp Shenzhen, we’ve helped a ton of people ship bulky items all over the world. The good news is that it doesn’t have to cost $200, the bad news is it will never be that cheap for small shipments. Here’s a basic rundown of the shipping options we’ve seen in Huaqiangbei.

Shipments under 20 kilos are considered very small shipments and there aren’t really any great options. If you’re visiting China the cheapest method is to lug it back in your suitcase.

shipping

The shipping calculator on our DEV site gives these prices for 8KG by DHL to the USA. These are roughly the prices you would get through any logistics company in Shenzhen. Ignore airmail, they won’t accept packages over 2KG. DHL China through a logistics agency is the best bet at $67, around $8.35 per kilo.

susi

At 10KG and 20KG there’s a price break for air freight through DHL/FedEx/UPS. Air freight takes a few days longer, and it has to be dropped off at the logistics agent. We’ve sent 6 hacker camps and hundreds of visitors to Susie Shipper in Huaqiangbei (pictured above) because they generally have someone who speaks English. There are dozens of similar air freight agencies in the market – just look for a garage full of boxes covered in packing tape.

For 20KG+ DHL air freight we’ve seen rates from $4.50/KG to Netherlands, and $5.50/KG to UK and USA through Susie Shipper.  At 100KG UPS air freight drops to $4/KG. Air freight is highly dependent on time of year. Around the October rush prices can be 10x higher, or the service may simply not be available at all.

Sea freight is a totally different beast and 8KG isn’t nearly enough. In our limited experience, Sea freight generally starts at $1000USD in fees alone. The actual cost of a 20 foot shipping container isn’t necessarily the greatest expense depending on destination. Sea freight has a cornucopia of associated charges that stack up fast. Bill of leading, document fees, dock fees, inspection, customs, loading, unloading, warehousing, pickup, delivery, plastic pallets, etc. There are some consolidators out there, but they generally want to ship a stuff that’s on a pallet too. 8KG doesn’t seem feasible to ship by sea.

Another thing to keep in mind is that bulky items are charged by dimension, not weight.

( Length * Width * Height ) / 5000 = dimensional weight

If the dimensional weight is greater than the measured weight, then the dimensional weight it used to calculate the shipping fees. It is frighteningly easy to run up against the dimensional weight.

This is the general frustration of China for foreign hobbyists. There are unimaginable, unbelievably cheap materials here, but unless you deal in giant volumes it is exceedingly hard to take advantage of those prices. Not only does shipping add up fast, but most suppliers have no ability to accept payments in foreign currency and exchange it into Chinese RMB.

There’s a much more fun option, one that’s only slightly more expensive than your initial shipping quote. Recently people have been finding flights from the US West Coast to China for around $400 round trip. Why not pop over, explore the markets, pack your suitcase with 23KG of goodies, and “pay” for the trip with the savings on shipping.

Site reorganization complete

via Dangerous Prototypes

Danger-sign

Sorry for the occasional outages over the past week. Most were during Chinese work hours, so hopefully not really noticed if you’re in the US or EU.

The whole site has been moved into GIT repositories and redeployed as virtual directories as outlined on the blog last week. It is much cleaner, better organized, and more resilient against bugs and hacks.

Today the blog moved to /blog/ to make way for the DEV site to be integrated here. All old posts, pages, and images have 301 Permanent Redirects to avoid breaking links at other sites and to preserve rank with search engines. DEV will probably be moved tomorrow if all goes well. All current DEV site orders will be processed without interruption or delay, even while the site is offline.

If you’re planning to get in any super cheap PCB or SLA 3D prints be sure to do it today. When we move the store to the main site prices will increase to match DirtyPCBs.com. Once the new store is running smoothly prices at DirtyPCBs will increase to encourage people to move to the new system. The DirtyPCBs scripts weren’t designed to handle the 10s of thousands of orders you’ve put through them and they’re starting to come apart at the seams.

The new forum theme is complete. phpBB will get a big update later this week. A new MediaWiki theme and upgrade are scheduled for next week.

If you notice any problems on the blog, forum, and wiki please report it through the contact form or here in the comments.

Phishing attempt in the forum and how to clean it up

via Dangerous Prototypes

Phishing_chart_Oct_2004_to_June_2005.svg

The forum, which runs on the popular phpBB3 software, was hit by a phishing attempt in the last few hours. Around 2000 private messages were sent by hundreds of newly registered users. It’s not clear how this automated attack got past our own spam hammer protection that bars new users from sending private messages. Apologies for the annoying messages and thanks to everyone for the reports.

Here is the SQL we used to clear out the scammers and their messages in case this post turns up search results.

SELECT *  FROM `phpbb_privmsgs` 
left join phpbb_users on author_id=user_id
WHERE `message_subject` LIKE '%We are deleting inactive accounts! Confirm your ac%'

First, backup your database!

Second, run this query to review what will get deleted. Replace “We are deleting inactive accounts! Confirm your ac” with the subject the phishers used. Could also be a line in the post or a URL.

DELETE `phpbb_users`, `phpbb_privmsgs`
FROM phpbb_privmsgs
left join phpbb_users on author_id=user_id
WHERE `message_subject` LIKE '%We are deleting inactive accounts! Confirm your ac%' AND user_id NOT IN (x,x,x,x)

If users responded to the phisher (ours had some choice words…) they will also get deleted because of the reply subject. Replace x in the NOT IN () clause with the user_id of any legit users that should not be deleted (or omit it if there are none).

Run the query to delete the messages and the users. There is no going back (you did a backup right?).

Image of global phishing incident reports by <a href=”htp://commons.wikimedia.org/wiki/User:Offnfopt” title=”User:Offnfopt”>Offnfopt</a>.