Your breast can start to swell as early as 1-2 weeks after conception because of hormonal changes and increased blood flow.