TPG Community

Ask, answer and talk about our products

Routing to AWS EC2 instances in SEA (Singapore) from Melbourne - something wrong

Highlighted
Level 1c

Hi folks,

 

fwiw a support ticket has been raised today, but thought I'd post to ask if anyone knows what is going on.

 

Background: I'm a software engineer with some network engineering background (enough to be dangerous Smiley Wink) and keep a close eye on latency on my network (being an avid gamer).

 

In the past week, something has been wrong - initially I thought it might be game bugs which are incorrectly determining latency into various AWS regions, but today I did further investigation, and have to put the blame squarely into TPG's lap, or the lap of one of the downstream partners - TPG need to fix this:

 

Below is a current trace of the route my packets are taking to reach an EC2 AWS gaming instance in Singapore (SEA - South East Asia - region). Notice the highlights... it's routing me through Los Angeles in the USA!!


To further diagnose, I activated my NordVPN VPN endpoint to a Melbourne location, and the trace went through the undersea cable via Perth directly to Singapore, so it's isolated to TPG based routing.

 

Note that in both cases, the EC2 server does not respond to ICMP packet latency requests, and so the trace drops one server back (but still in Singapore if one does a geolocation on the IP address).

 

fwiw, it only seems to affect ICMP packets - once I have established a connection to the Singapore endpoint and drop the VPN, all routing appears to be optimal (sub-100ms latency) - but ICMP ping requests are not routing correctly...

 

Any idea how this can be resolved?

 

(TPG routing):

tracert ec2-13-251-110-6.ap-southeast-1.compute.amazonaws.com

Tracing route to ec2-13-251-110-6.ap-southeast-1.compute.amazonaws.com [13.251.110.6]
over a maximum of 30 hops:

1 1 ms <1 ms <1 ms 192.168.0.1
2 2 ms 1 ms 1 ms 192-168-1-1.tpgi.com.au [192.168.1.1]
3 9 ms 9 ms 9 ms 10-20-25-29.tpgi.com.au [10.20.25.29]
4 9 ms 11 ms 10 ms 203-219-155-130.tpgi.com.au [203.219.155.130]
5 25 ms 22 ms 23 ms syd-gls-har-crt2-be-10.tpgi.com.au [202.7.173.17]
6 27 ms 22 ms 32 ms 203-221-3-67.tpgi.com.au [203.221.3.67]
7 169 ms 168 ms 169 ms las-b24-link.telia.net [213.248.95.232]
8 184 ms 188 ms 185 ms tata-ic-333371-las-b24.c.telia.net [80.239.128.215]
9 360 ms 361 ms 359 ms if-ae-2-2.tcore2.lvw-los-angeles.as6453.net [66.110.59.2]
10 527 ms 288 ms 288 ms if-et-53-2.hcore2.kv8-chiba.as6453.net [64.86.252.57]
11 374 ms 379 ms 373 ms if-ae-23-2.tcore1.svw-singapore.as6453.net [180.87.67.32]
12 362 ms 365 ms 361 ms 180.87.12.26
13 * * * Request timed out.
14 * * * Request timed out.
15 374 ms 384 ms 371 ms 52.93.10.57
16 370 ms 368 ms 369 ms 52.93.11.85
17 369 ms 368 ms 369 ms 52.93.11.94
18 362 ms 362 ms 362 ms 52.93.9.185
19 370 ms 369 ms 368 ms 52.93.10.101

 

(NordVPN routing):

tracert ec2-13-251-110-6.ap-southeast-1.compute.amazonaws.com

Tracing route to ec2-13-251-110-6.ap-southeast-1.compute.amazonaws.com [13.251.110.6]
over a maximum of 30 hops:

1 10 ms 9 ms 12 ms 10.8.8.1
2 11 ms 15 ms 11 ms 144.48.37.105
3 10 ms 10 ms 11 ms xe-4-0-0.cr2.me1.vic.au.coloau.net.au [103.52.117.104]
4 93 ms 92 ms 92 ms xe-4-0-2.cr1.p1.wa.au.coloau.net.au [45.112.247.163]
5 96 ms 93 ms 92 ms silicondesertinc.te0-6-1-18.br04.sin02.pccwbtn.net [63.217.25.194]
6 98 ms 92 ms 95 ms te0-6-1-18.br04.sin02.pccwbtn.net [63.217.25.193]
7 95 ms 92 ms 95 ms 63-218-107-42.static.pccwglobal.net [63.218.107.42]
8 * * * Request timed out.
9 * * * Request timed out.
10 153 ms 164 ms 155 ms 52.93.10.9
11 * * * Request timed out.
12 149 ms 149 ms 149 ms 52.93.11.8
13 148 ms 149 ms 148 ms 52.93.8.119
14 148 ms 147 ms 155 ms 203.83.223.33

14 REPLIES 14
Level 1c
Nice to see TPG responding to the more technical issues ... cheers! /sarcasm
Moderator

Hi @theexecutioner

 

Thanks for raising this to us. It is pretty obvious on your post that there is something wrong with the routing causing latency issue. This issue needs to be raised to our Engineering Team for further test and investigation.

 

We tried to use your community details to pull up the account unfortunately it shows multiple matching result.

 

Feel free to PM us your account details (Username/Customer ID together with the address on file).

 

In case you need a reference:

 

How do I private message (PM) in the community

 

Cheers!

 

 

 

Level 1c
Time's moved forward ... and the eng. team is apparently investigating. I'll post once I have an outcome. fwiw, Optus routes are fine ... so are NordVPN's ... but not TPG's .... so hopefully the eng. team can fix or trace to a downstream partner and get them to fix ... Cheers.
Moderator

We are glad to know that this issue is now being handled by our Engineering Team @theexecutioner. We'll keep an eye on this thread. Let us know how the case progress will go.

 

Cheers!

Level 1c

The Engineering team have fixed it apparently ...

 

Does that look fixed to you....? Smiley Mad Where did these bozo's learn to test...?

 

tracert ec2-13-251-110-6.ap-southeast-1.compute.amazonaws.com

Tracing route to ec2-13-251-110-6.ap-southeast-1.compute.amazonaws.com [13.251.110.6]
over a maximum of 30 hops:

1 3 ms <1 ms <1 ms 192.168.0.1
2 2 ms 1 ms 1 ms 192-168-1-1.tpgi.com.au [192.168.1.1]
3 11 ms 14 ms 13 ms nme-sot-dry-bras4-lo-20.tpgi.com.au [10.20.20.167]
4 16 ms 10 ms 12 ms nme-apt-bur-crt1-be10.tpgi.com.au [203.219.155.129]
5 32 ms 31 ms 30 ms syd-gls-har-crt1-be-10.tpgi.com.au [202.7.171.173]
6 32 ms 33 ms 28 ms syd-gls-har-int1-he0-3-0-2.tpgi.com.au [203.221.3.3]
7 174 ms 177 ms 177 ms las-b24-link.telia.net [213.248.95.232]
8 184 ms 185 ms 187 ms tata-ic-333371-las-b24.c.telia.net [80.239.128.215]
9 264 ms 266 ms 292 ms if-ae-2-2.tcore2.lvw-los-angeles.as6453.net [66.110.59.2]
10 272 ms 274 ms 277 ms if-ae-7-2.tcore2.svw-singapore.as6453.net [180.87.15.25]
11 265 ms 266 ms 267 ms 180.87.15.206

Moderator

Hi @theexecutioner,

 

Thanks for the update.

 

Please shoot me a PM with your TPG username or customer ID number. We'd like to look into it further.

 

Kind regards,

BasilDV

Level 1c

My original ticket has now been reopened. Anything you can do to help facilitate would be appreciated ... cheers.

Moderator

Hi @theexecutioner,

 

Thanks for notifying us. We will chase this up with our Engineering Team. Based on the latest log on file, we've seen that you have been in contact with one of our Engineers and advised that the Escalated Fault was re-opened, updates will be provided once update is available via phone call or SMS.

 

Let us know should you require further assistance.

 

Cheers!

 

 

Level 1c

Had a callback from the TPG team today ... "We don't have control over the routing..."

 

REALLY??

 

This is the answer from the network team? My goodness. While I accept that the TPG team can't directly make change if partners are incorrectly routing (assuming your primary routing table is correctly advertising optimal routes and it's being overriden downstream) I *absolutely* expect TPG to interact with industry partners to correct these type of problems downstream.

 

EDIT: To reiterate - I have had this same trace executed on an Optus cable connection and also when routed through NordVPN into Melbourne - BOTH route correctly, whereas TPG do not.

 

I'm thinking this may need to be raised up through an Industry Ombudsman ticket if I keep getting these type of answers from Network Support.

 

The clock is now ticking. I'm getting fed up with the run around...